Mert

Enterprise RAG — A practitioner’s build log | Post 1 of 6

There is a retrieval failure mode that does not show up in accuracy benchmarks: a system that finds the right document but returns it to the wrong person.

Most RAG evaluation frameworks measure whether the retrieved chunks are relevant to the question. Few measure whether those chunks should have been retrievable at all given who asked. In an enterprise context — where the same knowledge base holds HR policy, engineering runbooks, finance forecasts, and security incident reports — that gap is not a minor edge case. It is a fundamental design flaw.

I built Enterprise RAG specifically to treat access control as a first-class retrieval requirement, not an afterthought applied after the answer is generated.

The problem with post-retrieval filtering

The naive approach to document access control in a RAG system is to retrieve first and filter second: score all candidate chunks by relevance, then strip out the ones the user is not allowed to see before generating the answer.

This approach fails in two ways.

It leaks into the answer. A generative model given 20 chunks — including 5 restricted ones — can synthesize information from all 20 even if the restricted chunks are stripped from the citation list before the response is returned. The model has already read the finance forecast before you decided not to show it.

It provides false assurance. Citation filtering gives the appearance of access control without enforcing it in the part of the pipeline that matters. An audit of the response shows no restricted citations. But the answer content may reflect them.

The correct architecture applies access control before retrieval scoring. Unauthorized chunks are excluded from the candidate set entirely. They are never ranked, never passed to the generator, and never cited.

What silently leaks in a typical internal knowledge base

Consider a company running a single internal knowledge base with documents across four categories:

• HR and operations — visible to all employees
• Engineering runbooks — visible to engineers and above
• Finance forecasts and variance reports — visible to finance team and executives
• Security incident reports — visible to engineers and security team

A standard question like “What was the revenue variance in Q3?” asked by an employee role against a system with post-retrieval filtering may return an answer that reflects finance data — even if the finance document does not appear in the citation list. The system retrieved it, scored it, passed it to the generator, and then quietly removed it from the citations.

That is not a hypothetical. It is the predictable behavior of any system where retrieval and access control are separate pipeline stages.

The validation test that most teams do not run

Before I built anything, I defined the evaluation test that the system had to pass:

Ask the same question as two different roles. The answer content and the citation list should differ based on role. If an employee and a finance manager ask “What is the Q3 forecast variance?” and receive answers that contain the same information — regardless of whether the citations differ — the access control is not working.

The evaluation set in Enterprise RAG includes explicit forbidden document IDs per test case. The restricted_leak_count metric counts how many evaluation cases returned at least one forbidden document. For a system with correct pre-retrieval access control, that count should be zero.

The screenshot above shows this test passing: the employee role receives an answer grounded in publicly accessible policy documents, while the finance role receives an answer that additionally cites the restricted finance document. Same question. Different retrieval sets. No leakage.

What this changes operationally

The operational implication is that RAG deployment in an enterprise knowledge base requires a different validation standard than consumer or internal-tooling RAG.

Retrieval relevance is not sufficient. You need:

• A role model that maps document access to user identity
• Pre-retrieval filtering enforced before scoring
• An evaluation set that includes forbidden documents per role, not just expected documents
• A restricted_leak_count metric tracked alongside pass rate and citation coverage

Without all four, you cannot know whether your system is leaking restricted content. You can only know whether it is retrieving relevant content — which is a different and less important question in an enterprise security context.

Current limits

• The current implementation uses lexical retrieval with token cosine similarity scoring. Semantic or hybrid retrieval is a planned extension. Lexical retrieval is accurate enough for the validation workflow but does not match production semantic search quality.
• Role metadata is embedded in document front matter. Production deployments should derive role context from Entra ID or an OIDC identity provider, not request body parameters.
• The reference documents are synthetic. The evaluation set is calibrated for repeatable local validation, not a production-scale golden set.
• Multi-tenant isolation is a documented production consideration. The current implementation is single-organization.

Next engineering step

Run POST /eval/run against the seeded demo data and check the restricted_leak_count. If it is zero, access control is enforced. Then modify the retrieval pipeline to apply scoring before filtering and observe what changes in the evaluation output.

One question for you

If you queried your internal knowledge base with a restricted finance document in the index today, would your evaluation set detect whether that document’s content influenced the answer — or only whether it appeared in the citation list?

Next post: The architecture that puts access control before retrieval scoring, and why the order of operations is the entire design.

The first time I got a confirmed transaction on devnet, I felt great for about 10 seconds.

Then I thought: I have no idea what actually just happened.

I’d been treating transactions like a black box. Type address, type amount, hit send, done. But that stops working the moment something breaks — or the moment you try to build anything beyond a basic transfer.

So I went and figured it out. Turns out a Solana transaction is made of four things. Once you see what each one does, a lot of other stuff clicks into place.

The four parts

• Instructions — what you want to do
• Accounts — what data will be touched
• Signatures — proof of authorization
• Recent blockhash — a built-in expiry stamp

That’s it. Let’s go through them.

Instructions

An instruction is the action. It says: call this program, with these inputs.

For a basic SOL transfer, the instruction calls Solana’s System Program — a built-in program that handles things like moving SOL between wallets. You don’t deploy it, it’s just there.

The thing that surprised me: one transaction can hold multiple instructions. You could transfer SOL, create an account, and call another program — all in one shot. And it’s all-or-nothing. Either every instruction succeeds or the whole transaction fails. No partial results.

Accounts

In Solana, almost everything is an account — your wallet, a token balance, a deployed program. The whole network state is basically one giant table where every row has an address and some data.

When you write an instruction, you have to declare every account it will touch upfront — even ones you’re only reading. You also mark each one as writable or read-only, and whether it needs to sign.

This felt like extra work at first. But the reason is smart: Solana uses that list to figure out which transactions can run in parallel. No shared accounts = no conflict = process them at the same time. It’s a big part of why Solana can handle so many transactions per second.

The recent blockhash

Every transaction has to include a recent blockhash — a hash of a block that was produced recently on the chain.

It does two things.

First, replay protection. Without it, someone could take your signed transaction and submit it again later. The blockhash ties the transaction to a specific moment, so the network rejects anything that references an old one.

Second, expiry. A blockhash is only valid for about 150 blocks — roughly 60 to 90 seconds. After that window, the transaction is dead. You’d need a fresh blockhash and start over.

This is where Solana diverges from anything in Web2. A database transaction doesn’t expire. You can open one, walk away, come back. Solana transactions are more like a signed cheque with an expiry date on it — legitimate signature, but useless after the window closes.

Practical note: fetch the blockhash as late as possible, right before you send.

Signatures

Before a transaction is broadcast, it gets signed with your private key. Validators verify the signature against your public key before running anything.

If the transaction was tampered with after signing, the signature won’t match and it gets rejected.

Two things worth knowing:

Transactions can need multiple signatures. Some programs require both parties to sign. Multisig setups require a minimum number of co-signers.

The signature is the transaction ID. That long string in the Explorer URL? That’s not a separately generated ID — it’s the actual cryptographic signature. Every transaction ID on Solana is its own proof of authorization.

Go look at one on Explorer

Open any of your devnet transactions on Solana Explorer. You’ll see:

• Account Inputs — every account, labeled writable or read-only
• Instructions — each one expanded with its program and data
• Signatures — at the bottom of the page

Map what you see back to the four parts above. It lands differently when the transaction on screen is one you sent yourself.

The one thing to take away

A Solana transaction isn’t just a request. It’s a signed, time-limited, all-or-nothing bundle that declares its own authorization and expiry before it hits the network.

Once that settled in for me, a lot of things that seemed weird about Solana started making sense.

Part of my #100DaysOfSolana journey.

If you’ve tried prompting Hope AI, you know the first version is a production-grade application that can be used immediately. But a working app isn’t always the right app.

As you review the generated output, you may notice areas where the application isn’t aligned with your requirements or where boundaries and interfaces need adjustment. That’s a normal part of the process, and you shouldn’t have to start over to address it.

Hope AI’s output is designed to be refined in place, with components and contracts that stay consistent as you make changes. That stability lets you narrow the scope and improve the structure step by step, building forward instead of starting over

This article covers how refinement works in Hope AI. It explains how to improve output after the first version and how teams use this process to move toward review and release.

What is refinement in Hope AI?

Refinement is the process of aligning the generated structure with what you actually intend to build. It is how you make the structure clearer and more focused. This stage is where many teams lose momentum by jumping straight into implementation details, rather than clarifying what the product should do.

That approach tends to backfire because external tools often provide generic solutions that can clash with existing architectural decisions or introduce unnecessary complexity. The result looks more technical, but it’s harder to evaluate and extend.

Effective refinement works the other way around. It starts by narrowing the question. What behavior needs to change? Which feature is affected? How should the user experience differ after the change?

That kind of request gives Hope AI something concrete to work with. Once the behavior is clear, adjusting the structure becomes straightforward. You might split responsibilities, tighten interfaces or move logic into a more appropriate place, but those changes follow naturally from intent.

The important point is that refinement builds on what already exists. You are not replacing the system or re-specifying everything from scratch. The overall shape remains intact, while each pass makes the code easier to review, explain and move forward with.

How refinement with Hope AI differs from other AI builders

Many AI app builders treat updates as replacements. When a change is requested, the system regenerates large portions of the output, often resetting context and obscuring earlier structural decisions.

Hope AI follows a different approach.

Because components and contracts persist across iterations, changes are applied within existing boundaries rather than replacing the output completely. Context also accumulates as the system evolves and earlier decisions continue to shape what comes next. The table below highlights this difference.

Since structure and intent remain intact, developers can make focused adjustments that improve quality without destabilizing the system. Each change builds on what already exists, which is why refinement in Hope AI tends to strengthen earlier work rather than undo it.

That’s the foundation for the techniques below.

Techniques for refining Hope AI output

Here are some practical techniques to refine Hope AI output and prepare it for review.

Refinement usually involves:

• Narrowing component responsibility when boundaries blur
• Defining explicit contracts and interfaces
• Using tests to make expected behavior clear
• Respecting existing patterns unless the reason to break them is explicit
• Asking Hope AI to explain architectural decisions before changing them
• Aligning naming conventions for consistency
• Refactoring integration points to keep them flexible

Each of the techniques below expands on one of these moves and shows how to apply it without having to start over.

1.Narrow component responsibility when boundaries blur

When components have overlapping responsibilities, review becomes difficult. Mixed logic forces reviewers to sort through code just to understand intent. Splitting those concerns into focused pieces with clear boundaries makes the structure easier to follow.

// Before
    function UserProfile() {
      const handleLogin = async (credentials) => {};
      const handleUpdate = async (data) => {};

      return (
        <div>
          {!user ? <LoginForm onSubmit={handleLogin} /> : null}
          {isEditing ? <EditForm /> : <DisplayProfile />}
        </div>
      );
    }

    // After
    function ProfileDisplay({ user }) {
      return <div>{user.name}</div>;
    }

    function ProfileEditor({ user, onSave }) {
      return <form onSubmit={onSave}>...</form>;
    }

    function AuthManager({ onAuthSuccess }) {
      return <LoginForm onSubmit={handleLogin} />;
    }

Components that have a single responsibility are easier to review. Reviewers can review each part independently without having to sort through mixed logic.

2.Define explicit contracts and interfaces

Generic objects or unclear methods can lead to runtime errors. By defining clear contracts for data and component boundaries, teams can spot mismatches early and keep changes isolated.

// Before
    function UserForm({ onSubmit }) {
      const handleSubmit = (data) => {
        onSubmit(data);
      };
    }

    // After
    interface UserFormData {
      email: string;
      name: string;
      age: number;
    }

    function UserForm({ 
      onSubmit 
    }: { 
      onSubmit: (data: UserFormData) => Promise<void> 
    }) {
      const validate = (data: UserFormData) => {
        if (!data.email.includes('@')) {
          return { field: 'email', message: 'Invalid email' };
        }
      };
    }

Clear contracts help teams find integration issues during development. Reviewers can verify that components work together simply by examining the interface definitions.

3.Use test descriptions to clarify expected behavior

If test names are too vague, it’s hard to tell what the component does. Reviewers can’t verify that the code is correct by looking at generic test descriptions. Use test names that clearly describe the behavior you’re testing.

// Before
    describe('EmailValidator', () => {
      it('works', () => {
        expect(validate('')).toBe(false);
      });
    });

    // After
    describe('EmailValidator', () => {
      it('rejects empty email addresses', () => {
        expect(validate('')).toEqual({
          valid: false,
          error: 'Email is required'
        });
      });

      it('accepts valid email format', () => {
        expect(validate('[email protected]')).toEqual({
          valid: true
        });
      });

      it('trims whitespace before validation', () => {
        expect(validate('  [email protected]  ')).toEqual({
          valid: true
        });
      });
    });

Descriptive test names indicate how the code should work, so teams can infer the requirements from them.

4.Respect existing patterns unless the reason is explicit

Refinement can weaken a system when it introduces behavior that doesn’t line up with how the rest of the codebase already works.

Breaking a pattern can still be the right decision. What matters is whether the reason is stated explicitly in the request. When the business context is explicit, Hope AI can apply the change in a narrow way while preserving the rest of the system’s structure.

5.Ask Hope AI to explain architectural decisions

Sometimes, the generated structure reflects design trade-offs that aren’t immediately obvious. Without that context, it can be harder to evaluate how the architecture fits your project.

When something isn’t clear, ask Hope AI to explain the reasoning behind its choices and the trade-offs involved. This gives you a clearer view of how the design aligns with your requirements and where you might want to adjust scope or complexity as refinement continues.

6.Clarify naming conventions for consistency

When components and functions use different naming styles, it becomes harder to understand the code. Developers end up spending time on naming conventions rather than on logic. To avoid this, use consistent naming conventions so the codebase is easy to scan and understand. Consistent naming helps teams quickly spot component types, utilities and hooks without having to read through all the code.

7.Ask Hope AI to refactor integration points to avoid vendor lock-in

Components that interact with external systems benefit from clearly defined integration boundaries. You can ask Hope AI to refactor integrations using clear adapter interfaces, making it easy to swap out external services.

Putting vendor-specific details behind clear contracts helps keep your core components stable and makes it easy to swap out different implementations without changing the system’s core logic.

When to stop refining and start reviewing

You know you’re ready to review when further changes no longer meaningfully improve the generated structure. In practice, teams are ready to review Hope AI output when the following conditions are true:

• Each component has a clear responsibility that can be explained plainly, without qualifiers.
• Interfaces express intent directly, without relying on comments or implicit assumptions.
• Tests describe expected behavior clearly and fail for meaningful reasons.
• Changes to one component stay contained and don’t ripple into unrelated areas.
• The code feels ready to hand off to another engineer without additional context.

At this point, refinement has done its job. The structure is stable and the system can be evaluated and extended through normal peer review processes.

Wrapping up

In Hope AI, work begins with prompting. Right from your first prompt, you receive well-structured, production-ready code. The next step is refinement, where teams adjust the output to fit their workflows and prepare it for review.

If you remember only one thing about refining Hope AI output, make it structural consistency. Refinement works best when you follow the patterns already present in the system, where each feature owns its UI, data logic and API surface. Building within that structure keeps changes contained and maintenance straightforward.

Next, try using one or two of these refinement techniques on your current Hope AI project and see how quickly the output becomes ready for review.

AI development tools such as Cursor and Lovable make it possible to build working applications quickly, but that speed comes with a serious side effect.

Responsibilities that should remain separate often end up combined in the same components, with request handling, service calls, decision logic and data operations written together.

Teams that successfully deploy these AI-generated applications into production address those challenges through architectural separation, dividing the system into layers, each performing a specific role before passing the request along.

This article explains the three-layer architecture behind production-ready applications built with AI tools. It describes what each layer does, how requests move through them and which failures appear when those boundaries are missing.

The three-layer production architecture

AI-generated applications often run into problems when multiple responsibilities are combined. For example, issues can arise if a component manages authentication and also calls an AI service, if a request handler starts automated workflows or if a service writes to the database while interpreting AI output. These operational concerns should be kept separate.

Production-grade AI-generated applications are typically structured around three layers:

• Presentation layer – governs how requests enter the system

• Application layer – governs how application decisions are made

• Data layer – governs how data is stored and retrieved

The presentation layer governs system entry. Every request passes through authentication, input validation and rate limiting before reaching any application logic. Adversarial inputs and malformed payloads are also handled here before they affect internal services.

The application layer governs decisions. Application workflows run in this layer, and external services are integrated into those workflows. Responses from those services, including AI services, move through orchestration, validation checks and rule enforcement before any automated action occurs.

The data layer governs data persistence. It manages how application data is written, updated and retrieved across the system. Databases, storage systems and data access patterns operate in this layer, providing a consistent foundation for storing application state. Records of application activity, service responses and decision outcomes are also stored here so system behavior can be inspected and audited when needed.

Requests move through these layers sequentially, with each layer performing its checks and passing control to the next. The sections below describe each layer in detail, starting with the data layer, which should be designed before the application is built.

Layer 3 – The data layer

The data layer governs how application data is stored and how system activity is recorded. Building it early provides the persistence and traceability needed to recover from failures and understand how they occurred.

This layer is typically responsible for the following functions:

1.Data storage

The data layer manages how application data is written, updated and retrieved. Databases, storage systems and data access patterns operate here to keep application state consistent and available across the system.

2.Data pipelines

Data pipelines control how information enters and moves through the system. Inputs pass through ingestion paths that enforce schema validation, sanitize payloads, apply access permissions and record transformations as data flows between services. These controls protect data integrity while preserving a record of what entered the system and when.

3.Activity records

Applications that integrate external services generate additional system records alongside standard application data. Inputs sent to services, responses returned and the resulting system decisions are stored for auditing and debugging.

These records allow teams to reconstruct how a particular result was produced when investigating incidents or reviewing system behavior. They also provide the historical data that observability systems analyze to detect behavioral changes over time.

Layer 2 – The application layer

The application layer governs how decisions are made. Requests reaching this layer have already passed authentication and validation and are now processed by the application logic.

This layer typically handles the following concerns.

1.Orchestration

Orchestration manages how the application interacts with internal components and external services. It constructs requests, processes responses and handles operational concerns such as retries, timeouts and error handling.

By centralizing these interactions, orchestration prevents service failures or malformed responses from reaching users and keeps requests on a consistent execution path.

2.Rule enforcement

Application rules determine how system decisions are made. These rules enforce constraints such as approval thresholds, escalation policies, account tiers and workflow conditions. Placing these constraints inside the application layer prevents external service responses from directly controlling application behavior.

3.Feature flags

New behavior should be introduced gradually rather than deployed to all users at once.

Feature flags allow teams to control how functionality is rolled out by enabling changes for internal traffic first, expanding to limited user segments and eventually releasing to the full user base once system behavior remains stable.

This layer acts as the control center of the application. External services provide signals, while the application layer determines how those signals influence system behavior.

Layer 1 – The presentation layer

The presentation layer governs what enters the system. Every external request passes through it before reaching application logic, making it responsible for authentication, validation and request control.

This layer handles the following.

1.Authentication and access control

Requests must carry a verified identity, e.g., a Bearer token, before the system processes them. Role-based access control must also determine which operations each identity is permitted to perform. Without these controls, external requests can trigger system actions that cannot be traced to a specific user or workflow.

2.Input validation

User input must be validated before entering the system. Structured request schemas enforce predictable formats and prevent malformed payloads from reaching application logic. For applications that integrate AI capabilities, input validation also helps reduce the risk of prompt injection.

3.Rate limiting

Rate limiting protects the system from excessive traffic and resource exhaustion. A single unprotected endpoint under sustained load can quickly consume available capacity. Rate limits typically operate across several dimensions, including per-user quotas, endpoint throttling and adaptive controls that respond to system load.

4.Request and response formatting

Consistent request and response structures simplify processing across the system. When incoming requests follow predictable schemas, the application layer can evaluate them without handling arbitrary input shapes.

How the layers connect

The three layers provide operational safety only when requests pass through them in sequence. Systems that implement each layer but allow components to bypass boundaries recreate the same failure conditions that the architecture is meant to prevent. 

A request walkthrough illustrates how the layers interact under normal conditions.

Presentation layer

• A user submits a support ticket through the application interface.

• The request carries an authentication token that is validated by the identity service.

• Role-based permissions are checked for the requested operation.

• The request schema is validated against the expected format.

• The input is checked for malformed or unsafe content.

• The rate limiter verifies that the user has not exceeded their quota.

• The request is normalized into the expected structure before entering the application layer.

Application layer

• The orchestration component receives the request and coordinates the processing workflow.

• The application calls an external service to analyze the support ticket.

• The service returns a structured response describing the ticket category, priority level and suggested action.

• Application rules evaluate whether the suggested action is allowed based on policies such as approval thresholds, escalation rules and account tier.

• Feature flags determine whether the new automation behavior is enabled for this request.

• The application determines the final action and prepares the response.

Data layer

• The system stores the request payload and the resulting application state.

• Activity records capture the service response and the decision taken by the application.

• Data pipelines record how the request moved through the system for auditing and debugging.

• These records allow engineers to reconstruct how the system processed the request.

If an action triggers an incident days later, engineers can trace the full decision path through the logged request record.

Common architectural mistakes

Production failures often trace back to architectural shortcuts taken early in development. These problems usually appear when the responsibilities of the three layers are ignored or collapsed together.

1.Skipping presentation-layer controls

Some systems allow requests to reach application logic without proper validation. Authentication, request validation and rate limiting are either incomplete or missing entirely.

Without these controls, malformed inputs reach internal services, traffic spikes exhaust system capacity and requests cannot be tied to a specific identity. Problems that should have been stopped at the system boundary propagate throughout the application.

2.Placing application logic inside request handlers

Another common mistake is embedding orchestration, service calls and rule evaluation directly inside request handlers.

When this happens, the presentation layer and application layer collapse into a single component. Authentication, request parsing, service interaction and decision logic all run in the same execution path.

This structure makes the system difficult to maintain. Changes to one part of the workflow affect the entire request path, and failures become harder to isolate.

3.Allowing external services to determine system behavior

When applications return service responses directly to users or trigger workflows without applying application rules, those services effectively control system behavior. Incorrect outputs or unexpected responses propagate through the system without evaluation.

The application layer must remain the authority that determines which actions are allowed.

4.Failing to record system activity

Systems that do not store activity records become difficult to operate in production. Without records of inputs, service responses and decision outcomes, teams cannot reconstruct how the system processed a request. Incident investigations rely on guesswork and behavioral changes become difficult to detect. Operational visibility depends on the records maintained in the data layer.

5.Building rollback mechanisms after deployment

Rollback capabilities must be in place before the system reaches production. When configuration changes, service integrations or data transformations are not tracked, teams cannot isolate which change caused a failure. This increases incident duration and operational risk.

Closing out

AI development tools accelerate how quickly applications can be built, but that speed often introduces architectural shortcuts. As seen in this article, responsibilities such as request handling, service interactions, decision logic and data operations frequently end up combined in the same components.

Separating these responsibilities through a layered architecture restores that control. The presentation layer governs how requests enter the system, the application layer evaluates service responses and applies system rules and the data layer records the activity needed to monitor and recover from failures.

At Bit Cloud, this architectural separation forms the foundation for building and operating production AI systems. Teams that structure their systems this way gain the control and visibility required to run applications safely under real production conditions.