Mert

Semantic layers don’t fail because the technology is wrong. They fail because of design decisions made in the first two weeks — choices that seem reasonable at the time and create compounding problems for months afterward.

Here are the seven mistakes that kill semantic layer projects, and how to avoid each one.

Mistake 1: Defining Metrics in Multiple Places

What happens: Revenue is defined in a Tableau calculated field, a Power BI DAX measure, a dbt model, and a SQL view. Four sources of truth. None of them agree.

Why it’s common: Teams adopt new tools without migrating metric definitions. Each tool gets its own model. Over time, the definitions drift.

The fix: Every metric gets exactly one canonical definition in the semantic layer. All downstream tools query that definition. No exceptions. When someone needs Revenue, they query business.revenue, not their own formula.

This principle extends to AI agents. If your AI generates its own metric formulas instead of referencing the semantic layer, you’ve just added another source of truth — the least trustworthy one.

Mistake 2: Skipping the Bronze Layer

What happens: A data engineer creates a Silver view that joins raw source tables directly, mixing data cleanup (type casting, column renaming) with business logic (filters, calculations) in a single query. When the source schema changes — a column is renamed, a type is modified — the Silver view breaks.

Why it’s common: The Bronze layer feels redundant. It’s just a 1:1 mapping of the source. Why add a layer that doesn’t change anything?

The fix: The Bronze layer absorbs schema changes. When a source renames col_7 to order_date_utc, you update one Bronze view. The Silver and Gold views above it don’t change. This insulation is worth the tiny overhead of maintaining passthrough views.

Bronze views also standardize data formats. Timestamps normalized to UTC. Strings cast to consistent encodings. Column names made human-readable. This cleanup happens once, at the bottom of the stack, and every view above benefits.

Mistake 3: Using SQL Reserved Words as Column Names

What happens: A Bronze view exposes a column called Date. Now every downstream query must reference "Date" with double quotes. Analysts forget. AI agents don’t quote it at all. Queries break intermittently. Debugging is frustrating because the error messages are cryptic.

Why it’s common: Source systems often use generic names. Date, Timestamp, Order, Group, Role — all are SQL reserved words. Bronze views that don’t rename them propagate the problem to every consumer.

The fix: Rename early. In the Bronze layer, map Date to TransactionDate, Timestamp to EventTimestamp, Order to CustomerOrder. Use domain-specific prefixes that are unambiguous and never conflict with SQL keywords.

This small decision saves hundreds of hours of debugging across the life of the semantic layer. It also dramatically improves AI agent accuracy, since language models generating SQL rarely add appropriate quoting for reserved words.

Mistake 4: Building Without Stakeholder Input

What happens: A data engineering team builds 50 Silver views based on the database schema. They expose every table, every column, every possible metric. Business users look at the result, don’t recognize any of the terms, and go back to their spreadsheets.

Why it’s common: Data engineers understand the schema. They assume the schema structure maps to business needs. It usually doesn’t.

The fix: Start with a metric glossary co-created with stakeholders from Sales, Finance, Marketing, and Product. Ask them: What are your top 5 metrics? How do you calculate them? What decisions do they drive? Build the Silver layer around those answers, not around the database schema.

This step feels slow. It’s the fastest path to adoption. A semantic layer that uses business language and models business concepts gets adopted. A semantic layer that mirrors the database schema gets ignored.

Mistake 5: Treating Documentation as Optional

What happens: Views are created with no Wikis, no column descriptions, no Labels. The semantic layer works for the person who built it. Everyone else — analysts, AI agents, new team members — can’t figure out what the views mean.

Why it’s common: Documentation takes time. Deadlines are tight. Teams plan to “add documentation later.” Later never comes.

The fix: Make documentation part of the view creation process, not a follow-up task. At minimum, every view gets:

• A one-sentence description of what it represents
• Labels for governance (PII, Finance, Certified)
• Column descriptions for any non-obvious field

Modern platforms reduce this burden with AI-generated documentation. Dremio’s generative AI samples table data and auto-generates Wiki descriptions and Label suggestions. The AI provides a 70% first draft. The data team adds domain context for the other 30%.

Undocumented views are invisible to AI agents. If the Wiki is empty, the AI agent has no context to generate accurate SQL. Documentation isn’t just nice to have. It’s an accuracy requirement.

Mistake 6: Applying Security at the BI Tool Level Only

What happens: Row-level security is configured in Tableau so regional managers only see their region. Then an analyst opens a SQL client, queries the underlying table directly, and sees all regions. The security was enforced in the dashboard, not in the data.

Why it’s common: BI tools make it easy to apply filters and security rules. Data platforms require more setup. Teams take the easy path.

The fix: Enforce access policies at the semantic layer, not the BI layer. Row-level security and column masking should be applied on the virtual datasets (views). Every query path — dashboard, notebook, API, AI agent — inherits the same rules.

Dremio implements this through Fine-Grained Access Control (FGAC): policies defined as UDFs at the view level. A regional manager queries business.revenue and automatically sees only their region, regardless of how they access the data. No security gaps between tools.

Mistake 7: Trying to Model Everything at Once

What happens: The team commits to building a complete semantic layer covering every source, every table, and every metric. The project takes six months. By the time it launches, requirements have changed, stakeholder interest has waned, and half the views are out of date.

Why it’s common: Ambitious leaders want a “complete” solution. Data teams want to avoid rework. Neither wants to ship an incomplete layer.

The fix: Start with 3-5 core metrics that the organization actively debates (usually Revenue, Active Users, Churn). Build one Bronze → Silver → Gold pipeline per metric. Validate that the same question produces the same answer across two different tools.

Once those metrics are stable, expand incrementally. Add new sources, new views, new metrics — one at a time. Each addition is low-risk because the layered architecture isolates changes. A new Gold view doesn’t affect existing Silver views.

The fastest semantic layers reach 80% organizational coverage not by modeling everything up front, but by proving value quickly and expanding from momentum.

What to Do Next

Pick one mistake from this list. Check whether your semantic layer (or your plan for one) is making it. Fix that one thing this week. Then come back for the next one.

Try Dremio Cloud free for 30 days

Most of us use databases every day.

But at some point I started wondering:

• How does SQL actually work internally?
• How are queries parsed?
• How do joins work?
• What happens after a SELECT statement?
• How does persistence work under the hood?

So instead of only reading about databases, I decided to build one.

That project became Ark — a SQL-like relational database engine written entirely from scratch in C++.

Why I Built It

I wanted to understand the internals of database systems by implementing the pieces myself instead of relying on existing engines or parser generators.

The goal wasn’t to compete with production databases.

The goal was to learn:

• parsing
• query execution
• relational operations
• schema management
• persistence systems
• software architecture

Core Features

Ark currently supports:

• Handwritten tokenizer
• Recursive descent parser
• CRUD operations
• INNER / LEFT / RIGHT / FULL joins
• Aggregate functions (COUNT, SUM, AVG, MIN, MAX)
• ALTER TABLE
• LIKE pattern matching
• ORDER BY
• DISTINCT
• File persistence (SAVE / LOAD)
• Three-tier diagnostics system with exact line/column reporting

Everything is implemented manually:

• no external database libraries
• no parser generators

– no embedded SQL engines

Architecture

The execution pipeline looks roughly like this:

Query
  ↓
Tokenizer
  ↓
Parser
  ↓
Command Objects
  ↓
Execution Engine
  ↓
Storage Layer
  ↓
Persistence

The project is split into modular components:

• tokenizer
• parser
• execution engine
• diagnostics
• storage/persistence

Example Query

CREATE TABLE employees (
    id INT,
    name STRING,
    salary DOUBLE
);

INSERT INTO employees VALUES
    (1, "Alice", 95000.0),
    (2, "Bob", 72000.0);

SELECT * FROM employees
WHERE salary > 80000.0;

One of the Hardest Parts

One of the most interesting challenges was implementing joins and schema evolution.

Handling:

• ALTER TABLE
• adding/dropping columns
• persistence consistency
• join execution

became much more complicated than I initially expected.

Parser correctness and diagnostics also took a surprising amount of effort.

What I Learned

Building Ark taught me a lot about:

• how parsers actually work
• query execution pipelines
• relational database concepts
• software architecture
• debugging complex state systems
• designing diagnostics/error reporting

It also gave me a much deeper appreciation for real database engines.

GitHub

GitHub Repository:
https://github.com/kashyap-devansh/Ark

I’d genuinely appreciate feedback from people interested in:

• databases
• systems programming
• parsers
• compilers
• C++

Especially suggestions for improving the architecture or query engine.

Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptions that nobody has time to fix. The problem isn’t motivation. It’s that manual documentation doesn’t scale.

A self-documenting semantic layer changes the equation. Instead of asking humans to describe every column in every table, the platform generates descriptions automatically, suggests governance labels from data patterns, and propagates context through the view chain. Documentation becomes a byproduct of building the semantic layer, not a separate project.

The Documentation Problem Nobody Solves

Industry surveys consistently find that 70% or more of enterprise data assets are undocumented or poorly documented. The result: analysts spend 30-40% of their time searching for data and trying to understand what it means before they can start analyzing it.

This isn’t just a productivity problem. Undocumented data is a governance risk. A column named status with values 0, 1, 2, and 3 could mean anything. An analyst guesses. An AI agent guesses worse. Nobody verifies. The wrong assumptions get baked into dashboards that drive business decisions.

Data teams respond with documentation sprints. They burn a week writing Wiki pages for their top 50 tables. Two months later, half the descriptions are outdated because schemas have changed. The cycle repeats.

What Self-Documenting Actually Means

A self-documenting semantic layer generates and maintains documentation with minimal human effort. Three mechanisms work together:

AI-generated descriptions: The platform samples data in a table and generates human-readable descriptions for each column and the table itself.

Automated label suggestions: The platform analyzes column names, data types, and value patterns to suggest governance labels (PII, Finance, Certified).

Metadata propagation: When a Silver view references a Bronze view, column descriptions flow downstream automatically. Documentation written once at the Bronze level appears everywhere the column is used.

Human oversight is still essential. AI provides a 70% first draft. Data engineers add the domain-specific context that only they know: business rules, edge cases, known data quality issues. The point isn’t to eliminate human documentation. It’s to eliminate the blank page.

AI-Generated Descriptions

Modern semantic layer platforms can sample a table’s data and generate meaningful descriptions automatically.

Consider a column named cltv in a table called customers. The AI samples values (1200.50, 3400.00, 780.25), examines the column name and table context, and generates:

cltv: Customer Lifetime Value in USD. Represents the total revenue attributed to this customer from their first purchase to the current date, excluding refunded transactions.

Not every generated description will be this precise. But most are useful enough to replace the current state: an empty description that tells the analyst nothing.

More examples:

• A column with values “US”, “UK”, “DE” → “ISO 3166 alpha-2 country code for the customer’s billing address”
• A DATE column named created_at in a subscriptions table → “Date the subscription was created”
• A FLOAT column named mrr → “Monthly Recurring Revenue in the account’s base currency”

Automated Label Suggestions

Labels categorize data for governance and discovery. Manually tagging every column in a data warehouse with hundreds of tables is impractical. AI-based label suggestion makes it manageable:

• Columns containing email-like patterns (text with @ symbols) → suggested label: PII
• Columns with phone number patterns → suggested label: PII
• Columns named price, total, amount, revenue → suggested label: Finance
• Columns in tables marked “Certified” → suggested label propagated to downstream views

Dremio’s approach combines these suggestions with human approval. The AI proposes labels. A data engineer reviews and accepts or rejects. Over time, the catalog fills up with accurate, useful labels without dedicated labeling sprints.

Metadata Propagation Through Views

In a well-designed semantic layer, documentation shouldn’t need to be written more than once. The Bronze-Silver-Gold view architecture creates a natural propagation path:

1. Bronze layer: Document the CustomerID column as “Unique identifier for the customer, sourced from the CRM system.”
2. Silver layer: A Silver view references CustomerID. The description propagates automatically. No re-documentation needed.
3. Gold layer: An aggregated Gold view groups by CustomerID. The description carries through.

This propagation is especially valuable for join columns, filter columns, and commonly used dimensions that appear in dozens of views. Write the description once at the source, and it follows the column everywhere.

How This Reduces Toil

The impact on data team productivity is measurable:

The net effect: documentation coverage goes from 30% (what the team could manage manually) to 80-90% (AI baseline + human refinement). The team spends hours instead of weeks on documentation. And the documentation stays current because the AI can re-scan when schemas change — flagging outdated descriptions instead of waiting for someone to notice.

For AI agents, this improvement is material. A richer, more accurate semantic layer means the AI generates better SQL, hallucinates less, and requires fewer corrections. Self-documentation isn’t just a productivity feature. It’s an AI accuracy feature.

What to Do Next

Pick your most-used table. Open it in your data platform. How many columns have descriptions? How many have governance labels? If the answer is “not many,” calculate how long it would take to document the entire table manually. Then consider a platform that does 70% of that work for you.

Try Dremio Cloud free for 30 days

The Adopter is a company that uses an open source technology—it may very likely be a core part of their internal infrastructure—and they engage externally with that project’s community. They’re not trying to run the project. They’re not selling something based on it. They’re users who have chosen to be vocal.

If you’ve ever been to a conference and seen a talk titled something like “How ${BIG_COMPANY} Uses ${OSS_PROJECT} at Scale,” you’ve seen the Adopter model in action. Most consumer companies that show up at open source conferences fall into this category.

The factor profile

Where does the Adopter sit on the four diagnostic factors?

Commercial dependency: Low.

For the most part, the Adopter’s revenue doesn’t come from this project. In many cases, if they so choose, they could likely replace this component of their infrastructure without any of their customers noticing.

Project maturity: Established.

Adopters typically engage with more established, known projects rather than emerging ones. If they’re trusting a project enough to be embedded in their infrastructure, they want to know it’s stable.

Ownership: User level.

The Adopter is an end-user. Occasionally, they will engage with the project directly; for example, they might file issues, submit the occasional PR, or participate in community channels. But they’re not maintainers. They’re not steering the ship. And they don’t intend to.

Strategic intent: Reputation and/or recruiting.

They want the community to know who they are, what they’re building, and, frankly, they want talented engineers who are familiar with the open source project to want to come join their engineering teams.

Why companies end up here

The Adopter model often starts as a grassroots effort. It’s not a top-down strategic initiative from the C-suite. It’s a team of engineers who bet on a technology, built something meaningful with it, and realized their team could benefit from being visible in that project’s community. And it grows from there.

The motivations are usually some combination of:

• Recruiting Engineering talent is competitive. When your team gives a talk at a project’s conference about the hard problems they solved at scale, that’s a magnet for engineers who want to work on similar challenges. It’s more authentic than a careers page and more targeted than a job posting.
• Reputation Being known in a community gives your company credibility. Other companies want to hear what you’re building and how you’re solving problems. That recognition compounds over time and may naturally lead to invitations to speak, collaborate, and participate.
• Protecting a dependency This one’s less glamorous, but it’s still very real. If your infrastructure depends on a project, you have a vested interest in that project staying healthy. Engaging with the community—even just by providing production feedback and use cases—helps ensure the project evolves in ways that serve your needs.

This is an archetype I lived during my time as a software engineer at Bloomberg. After extensive research, my team had made the decision to use Kafka Streams for processing market data, and so began our foray into the Apache Kafka® community. Our manager encouraged us to give talks—for visibility in the community, for recruiting, and to build Bloomberg’s reputation as a company doing cutting-edge work with streaming technologies.

That grassroots, team-driven origin is typical of the Adopter model.

Tactics: what the work actually looks like

The Adopter’s playbook is the most straightforward of the four archetypes:

Attend and present at community events. Show up to the conferences, meetups, and community days for the project you use. Don’t just attend—submit talks. Share what you’re building. The bar isn’t “we did something nobody else has done.” The bar is “here’s what we learned using this thing in production.”

Write about your experiences. Blog posts, case studies, architecture breakdowns. Technical content that shares the patterns, challenges, and decisions you’ve made with the technology. This content is genuinely valuable to the community—it helps other users, validates the project’s approach, and sometimes surfaces edge cases the maintainers hadn’t considered.

Engage in community channels. Answer questions in Slack, comment on GitHub issues with your real-world experience, provide feedback on proposals. You don’t need to contribute code to be a valuable community member. Showing up with production context is a contribution in itself.

[Optional] Provide financial support. Some Adopters sponsor the project’s foundation or events. This is legitimate and often appreciated—but a note of caution: once money enters the picture, internal stakeholders may start asking about ROI in ways that push the engagement from genuine to performative. Be aware of that pressure.

The selfishness is the feature

I’ll be honest—there’s a degree of selfishness in the Adopter model. You’re doing this for recruiting. For reputation. For protecting a dependency you rely on.

And that’s okay.

Because the community benefits too. Real-world production use cases are gold for an open source project. When you stand up at a conference and say “we run this at scale and here’s what we learned,” that’s valuable to everyone. It validates the project. It helps other users avoid pitfalls. It gives maintainers signal about how the software is actually used.

The selfishness and the community benefit aren’t in tension. They’re aligned. That’s what makes this model work. You get what you need (visibility, hires, dependency health) and the community gets what it needs (production feedback, real use cases, engaged participants).

Measuring success

How do you know your Adopter engagement is working? There are two layers here.

Homegrown metrics

Since the Adopter’s goals are often self-interested, many companies in this position track their own internal signals:

• Content reach. Are people engaging with your talks and blog posts? Are your conference submissions getting accepted?
• Recruiting pipeline influence. Can you attribute inbound engineering interest to your community presence? Are candidates mentioning your talks or posts in interviews?
• Community recognition. Are you being invited to participate? Do people in the community know your company’s name and associate it with expertise?

CHAOSS metrics for the Adopter

If you’re not familiar, the CHAOSS project is an effort to better understand open source projects and communities through rigorous, standardized metrics models for measuring community health. For the Adopter, two models are particularly relevant:

• Business Readiness. This model helps you assess whether the project you depend on is healthy enough to keep relying on:

  • Defect Resolution Duration. Are bugs getting fixed in a reasonable timeframe? If defects linger, that’s a risk signal for your internal use.
  • Contributor Absence Factor. What’s the bus factor? Could this project survive if key maintainers left? If the answer is no, you might want to increase your engagement (or have a contingency plan).

• Project Awareness. This model tracks how well-known and actively engaged-with a project is:

  • Organizational Diversity. Is the project healthy and supported by multiple organizations? Or is it dangerously dependent on one company? (For the Adopter, this is a health check on your dependency, not something you’re trying to influence directly.)

Key Insight For the Adopter, these metrics are about monitoring risk and validating your choice. You’re asking “Is this project healthy enough to rely on?” not “Are we growing the community?” That distinction matters because the same metric can mean something very different in other archetypes.

Anti-patterns to watch for

Even in the most straightforward archetype, things can go wrong:

Talking without contributing. If your company only takes the stage but never participates, e.g. files issues, provides feedback, or engages in community channels, you’ll eventually be seen as a tourist. You don’t need to commit code, but you do need to participate beyond the spotlight moments.

Treating it purely as marketing. The moment your “community engagement” starts feeling like a brand campaign—messaging that’s too polished, product placement, no genuine technical depth—the community sees through it as inauthentic. Adopter advocacy works because it’s engineers sharing engineering work. The second it becomes marketing-driven, it loses credibility.

Ignoring project health. If you benefit from the project but never check on its sustainability—never look at whether maintainers are burning out, whether the bus factor is dangerously low, whether the project is losing contributors—you’re freeloading. That dependency risk will catch up with you.

Expecting influence without earning it. Being a high-profile user doesn’t automatically give you a seat at the governance table. Some Adopters get frustrated that their “big company voice” doesn’t translate into immediate bug fixes or project direction. That’s not how this works. Influence is earned through contribution, not brand weight.

Examples in the wild

• Most consumer tech companies at open source conferences — the Ubers, the Netflixes, the Walmarts giving talks about their streaming pipelines, their infrastructure at scale, their internal tooling built on open source.
• Any company whose engineering blog regularly features open source projects they use internally—this is Adopter advocacy in written form.

The Adopter in context

The Adopter model is often undervalued because it looks simple: show up, talk about your stuff, go home. But done well, it serves a critical function in the open source ecosystem by providing social proof, real-world validation, and production signal that projects need to grow.

And for the company doing it? It’s one of the highest-ROI forms of developer engagement you can do. The investment is relatively low (engineers sharing work they already did), the alignment between self-interest and community benefit is natural, and the risks are minimal compared to the other archetypes.

Just remember: the playbook is non-transferrable. What works here—showing up as a user, sharing experience, recruiting through visibility—will not work if your company’s situation changes. If you start contributing heavily enough to become a steward, or if your commercial relationship to the project deepens, you’ve drifted into a different archetype. And you’ll need a different approach to continue showing up credibly.

More on that in the final post of this series.

Next up: The Champion—what happens when your company invests heavily in a project that isn’t core to its main business. Follow along so you don’t miss it.

I used to think bundle optimisation was someone else’s problem. I’d write code using convenient namespace imports like import * as utils from './utils', run npm run build, and ship whatever came out. My bundles kept growing: 200KB, 300KB, 450KB, but I assured myself it was fine. After all, browsers were getting better, internet connections were faster, and devices were becoming more powerful.

Then I tested my 450KB utility library on a 3G connection. Four seconds to download. Lighthouse gave me an embarrassing performance score. That’s when I learned I was shipping 60% unused code.

This article covers what I discovered about tree-shaking in Vite, the mistakes I was making, and how I fixed them.

What Tree-Shaking Actually Does

Tree-shaking is dead code elimination for ES modules. When you use import and export, you create a static dependency graph. Vite (via Rolldown) traces through this graph and removes exports that aren’t being consumed. Basically, tree-shaking only works when the bundler can prove code is unused. If you give it ambiguous signals, it keeps everything safe.

The Anti-Patterns I Was Using

Take a utility component called dashboard.utils.ts as a real example. It exports eight standalone functions: hasActiveFilters, mapApplicationToTableData, resetPagination, updateFilterState, updatePagination, mapJobStats, normalizeFilter, and buildPagination. I only needed one, but here’s how I used to import it

Vite can statically analyse and determine that only resetPagination is accessed here, and drop the rest, but that’s the bundler doing you a favour, not you writing intentional code. You’ve imported the entire module and handed the cleanup responsibility to your build tool. It works until it doesn’t.

And it stops working the moment you do something like this:

Once the namespace object is passed to console.log or spread into another structure, the bundler can no longer prove at build time which properties will be accessed at runtime. It has no choice but to keep all eight exports: hasActiveFilters, mapJobStats, buildPagination, and everything else, just in case.

This is a trap because the first version looks harmless. It gets past your linter, the build succeeds, and you move on. Then three weeks later, someone adds a debug log, passes the namespace to a utility, and suddenly your bundle is carrying dead weight you didn’t notice.

The “Just in Case” Import Pattern

Eight dashboard utility functions imported. One actually used. Seven along for the ride on every page load.

The Fixes I Implemented

Named Imports Only

This provides a clear signal to the bundler: only resetPagination is needed. mapJobStats, buildPagination, normalizeFilter, and the rest can be safely removed from the final output.

Switching from namespace imports and over-importing to precise named imports was the single biggest improvement.

Pure Function Annotations

I noticed /* @__PURE__ */ comments in my build output. These tell the minifier a function call has no side effects, meaning if the return value is never used, the entire call can be safely removed.

In the snipper above, both resetPagination and buildPagination just take input, compute a value, and return it. No HTTP calls, no mutations, no DOM access. They have zero side effects. That’s exactly what Vite looks for when deciding whether to annotate a call as pure:

Strategic Dynamic Imports

Imagine your project grows and you have a helpers/ directory with multiple utility files, date.helper.ts, dashboard.utils.ts, and others. The temptation is to import them all eagerly:

Every page component, all its dependencies, and all its templates are now bundled together and loaded on the first request, regardless of which page the user is actually visiting.

Instead, lazy-load each page so that it becomes its own separate chunk:

A user who only visits the dashboard never downloads the profile management or jobs page. The router calls loadPage('dashboard') on navigation, and everything else stays unloaded until it’s needed.

My Current Practices

1. Named imports only
No import * as unless necessary.

2. Don’t pass namespace objects to functions
Log specific properties, not the whole namespace.

3. Dynamic imports for optional features
If it’s not needed for the initial render, lazy-load it.

4. Review dependencies
Replace non-tree-shakable libraries when possible.

Key Takeaway

Vite’s tree-shaking is powerful, but it only works with your cooperation. Write code that gives the bundler clear signals about what’s actually used.

Check your bundle. You might be shipping more dead code than you think.