If you’ve spent any time in a manufacturing environment, you know the dirty secret nobody talks about in conference rooms: the gap between what the ERP system thinks is happening on the shop floor and what’s actually happening is often enormous. Orders that are “in process” for three days. Quality defects discovered at final inspection that could have been caught at operation two. Machine downtime that gets logged retroactively—if it gets logged at all.

This is the problem that SAP MES integration with PP/QM is designed to solve. And after years of designing and delivering these architectures, I can tell you: when it works well, it genuinely transforms manufacturing operations. When it’s done poorly, you end up with a system nobody trusts and a shop floor that goes back to paper.

In this article, I’ll walk you through the architectural decisions, integration patterns, and real-time monitoring strategies that make the difference between a successful MES-SAP integration and an expensive failure.

What We’re Actually Trying to Solve: The Shop Floor Data Problem

Before we talk architecture, let’s be honest about the problem. Most manufacturing companies have SAP S/4HANA (or ECC) handling their production orders, quality inspection lots, and material movements. What they typically lack is real-time feedback from the shop floor. The MES system—whether that’s a third-party product like Siemens Opcenter, Rockwell Plex, or a custom-built solution—sits in the middle, capturing machine signals, operator inputs, and process data that the ERP was never designed to collect.

The integration challenge is bidirectional:

• Downward (SAP → MES): Production orders, routing operations, BOM components, quality inspection plans, and scheduling data need to flow to the shop floor system so operators know what to build and how.

• Upward (MES → SAP): Confirmations, actual quantities, quality results, scrap declarations, and material consumption need to flow back so the ERP reflects reality.

The real-time requirement is what makes this hard. A batch upload once an hour might have been acceptable in 2005. Today, with lean manufacturing targets and customer commitments that require hour-by-hour production visibility, you need confirmations flowing back within seconds or minutes of an event occurring on the floor.

Core SAP Objects You Need to Understand

Before designing any integration, make sure you’re fluent in the SAP side of this equation. Here are the objects that matter most:

Production Orders (PP)

• AUFK: Order header data (status, plant, MRP controller)

• AFKO: Order header for PP orders (scheduling dates, quantities)

• AFPO: Order items

• AFVC: Operations within the order

• AFVV: Quantities and times for operations

• RESB: Reservation/component requirements

Quality Management (QM)

• QMEL: Quality notifications

• QALS: Inspection lots

• QASE: Inspection characteristics

• QAMR: Inspection results (mean values)

• QAMV: Inspection results (single values)

Understanding these tables isn’t optional. When your MES team tells you “the confirmation isn’t working,” you need to know exactly which BAPI is being called, which fields are required, and which database tables to check for debugging. Ignorance of the underlying data model is where most integrations go wrong.

Architecture Patterns: Choosing the Right Approach

I’ve seen three main architectural patterns used for MES-SAP integration. Each has its place, and choosing the wrong one for your context is a mistake you’ll be living with for years.

Pattern 1: Direct BAPI/RFC Integration

The oldest and most common approach. The MES calls SAP BAPIs directly via RFC connection. For production confirmations, the primary BAPI is BAPI_PRODORDCONF_CREATE_TT. For goods movements, it’s BAPI_GOODSMVT_CREATE.

When it works well: Simple, low-volume scenarios with a single MES system and a stable SAP landscape.

When it fails: Network latency kills you in real-time scenarios. If the MES has to wait for a synchronous RFC response before allowing an operator to proceed, and SAP is having a slow moment, you’ve just created a bottleneck on the production line. I’ve seen this exact scenario halt a production line during a critical period. Not a good day.

Pattern 2: Message-Based Integration via Middleware

The MES publishes events to a message broker (Apache Kafka, RabbitMQ, or SAP Integration Suite), and an integration layer consumes those messages and pushes them to SAP asynchronously. For the reverse direction, SAP IDocs or change pointers trigger outbound messages.

This is the pattern I recommend for most modern implementations. The decoupling is essential. The shop floor system doesn’t need to wait for SAP. If SAP is temporarily unavailable, messages queue up and process when connectivity is restored. You get retry logic, dead-letter queues for failed messages, and monitoring out of the box.

If you’re building event-driven architectures in your SAP landscape, the patterns I discussed in SAP BTP Event Mesh: Building Resilient Integrations That Actually Scale apply directly here—especially the guidance around guaranteed delivery and consumer group management.

Pattern 3: IoT-First with Edge Processing

The most modern approach: machine signals are collected at the edge (using OPC-UA, MQTT, or similar protocols), processed locally to generate business events (“operation completed,” “quality threshold exceeded”), and then those business events are forwarded to both the MES and SAP. The MES becomes a consumer of events rather than the sole source of truth.

This is the direction the industry is moving, particularly in Industrie 4.0 contexts. It enables real-time anomaly detection without requiring SAP to be in the critical path of machine operation.

Practical Implementation: Production Order Confirmation Flow

Let me walk through the most common use case: confirming a production order operation from the MES back to SAP. Here’s the ABAP side of that equation—a wrapper function that handles the BAPI call with proper error handling.

"""
Wrapper for MES Production Order Confirmation
Calls BAPI_PRODORDCONF_CREATE_TT with proper error handling
Prerequisite: Valid production order exists in AUFK with operations in AFVC
"""
CLASS zcl_mes_prod_confirmation DEFINITION PUBLIC FINAL CREATE PUBLIC.

  PUBLIC SECTION.
    TYPES:
      BEGIN OF ty_conf_result,
        success     TYPE abap_bool,
        message     TYPE string,
        conf_number TYPE rueck,
      END OF ty_conf_result.

    METHODS confirm_operation
      IMPORTING
        iv_order_number TYPE aufnr
        iv_operation    TYPE vornr
        iv_conf_qty     TYPE menge_d
        iv_scrap_qty    TYPE menge_d
        iv_actual_work  TYPE ae_arbmng  "Actual labor hours"
        iv_posting_date TYPE budat
      RETURNING
        VALUE(rs_result) TYPE ty_conf_result.

  PRIVATE SECTION.
    METHODS build_conf_header
      IMPORTING
        iv_posting_date TYPE budat
      RETURNING
        VALUE(rs_header) TYPE bapi_bus2105_conf_head_create.

    METHODS handle_return_messages
      IMPORTING
        it_return TYPE bapiret2_tab
      RETURNING
        VALUE(rv_success) TYPE abap_bool
      CHANGING
        cv_message TYPE string.

ENDCLASS.

CLASS zcl_mes_prod_confirmation IMPLEMENTATION.

  METHOD confirm_operation.
    DATA:
      ls_header     TYPE bapi_bus2105_conf_head_create,
      lt_timetickets TYPE TABLE OF bapi_bus2105_conf_tt_create,
      ls_timeticket  TYPE bapi_bus2105_conf_tt_create,
      lt_return     TYPE TABLE OF bapiret2,
      lv_conf_num   TYPE rueck.

    "Build the confirmation header"
    ls_header = build_conf_header( iv_posting_date ).

    "Populate the time ticket (operation details)"
    ls_timeticket-orderid    = iv_order_number.
    ls_timeticket-operation  = iv_operation.
    ls_timeticket-yield      = iv_conf_qty.
    ls_timeticket-scrap      = iv_scrap_qty.
    ls_timeticket-work_actual = iv_actual_work.
    ls_timeticket-work_unit  = 'H'.  "Hours"
    ls_timeticket-fin_conf   = abap_false.  "Not final confirmation"
    ls_timeticket-postg_date = iv_posting_date.

    APPEND ls_timeticket TO lt_timetickets.

    "Call the BAPI"
    CALL FUNCTION 'BAPI_PRODORDCONF_CREATE_TT'
      EXPORTING
        headdata           = ls_header
      TABLES
        timetickets        = lt_timetickets
        return             = lt_return.

    "Handle results"
    rs_result-success = handle_return_messages(
      EXPORTING it_return  = lt_return
      CHANGING  cv_message = rs_result-message
    ).

    IF rs_result-success = abap_true.
      "Commit the work - critical step often missed!"
      CALL FUNCTION 'BAPI_TRANSACTION_COMMIT'
        EXPORTING
          wait = abap_true.  "Wait for async update tasks"
    ELSE.
      "Roll back on any error"
      CALL FUNCTION 'BAPI_TRANSACTION_ROLLBACK'.
    ENDIF.

  ENDMETHOD.

  METHOD build_conf_header.
    rs_header-postg_date = iv_posting_date.
    rs_header-conf_text  = 'MES Automatic Confirmation'.  " Audit trail"
  ENDMETHOD.

  METHOD handle_return_messages.
    rv_success = abap_true.
    DATA(lv_errors) = VALUE string_table( ).

    LOOP AT it_return INTO DATA(ls_return).
      CASE ls_return-type.
        WHEN 'E' OR 'A'.
          rv_success = abap_false.
          APPEND |{ ls_return-id }-{ ls_return-number }: { ls_return-message }|
            TO lv_errors.
        WHEN 'W'.
          "Log warnings but don't fail"
          MESSAGE s001(zmesint) WITH ls_return-message.
      ENDCASE.
    ENDLOOP.

    IF lv_errors IS NOT INITIAL.
      cv_message = concat_lines_of( table = lv_errors sep = | | ).
    ENDIF.
  ENDMETHOD.

ENDCLASS.

A few things worth highlighting in this code:

• The BAPI_TRANSACTION_COMMIT with wait = abap_true is critical. Without it, the update runs asynchronously and you can’t be certain the data is committed before you respond to the MES. I’ve debugged production issues that boiled down to this missing parameter.

• Always distinguish between Error/Abort messages and Warnings. Warnings shouldn’t block processing—log them for monitoring but let the transaction proceed.

• The rollback on error is non-negotiable. Partial confirmations create data integrity nightmares that are very hard to clean up.

Quality Inspection Results: Closing the Loop

The QM side of MES integration is often underestimated. Most teams focus on production confirmations and treat quality as an afterthought. This is a mistake.

When an operator performs an in-process quality check at a workstation, those results need to flow back to SAP’s inspection lot (QALS) and update the inspection characteristics (QASE). The BAPI for recording inspection results is BAPI_INSPOPER_RECORDRESULTS.

The architectural consideration here is about usage decision triggering. In SAP QM, the usage decision is the formal acceptance or rejection of an inspection lot. In an MES-integrated scenario, you have a choice:

• Let the MES trigger the usage decision automatically based on measurement results vs. tolerances

• Record results in SAP and let the QM team make the usage decision manually in QA11

• Use a hybrid: auto-approve lots where all results are within tolerance, flag exceptions for manual review

Option 3 is almost always the right answer. Full automation sounds appealing but removes human judgment from critical quality decisions. Pure manual processes defeat the purpose of integration. The hybrid approach gives you efficiency where it’s safe and human oversight where it matters.

Real-Time Monitoring: What Good Looks Like

The goal of MES-SAP integration isn’t just data synchronization—it’s operational visibility. Here’s what a mature implementation should provide:

Key Performance Indicators to Track

• Confirmation lag: Time between operation completion (MES timestamp) and SAP confirmation posting. Target: under 2 minutes for real-time scenarios.

• Integration error rate: Percentage of MES messages that fail to process in SAP. Target: below 0.5%.

• Quality result turnaround: Time from inspection completion to usage decision in SAP.

• Schedule adherence: Comparing planned vs. actual operation times using AFVV data.

Building a Custom Monitoring Dashboard

Don’t rely solely on standard SAP transactions for monitoring your integration. Build a custom ABAP CDS view that joins your integration log table with production order data. If you want to go deep on how CDS views can power real-time reporting scenarios like this, the techniques covered in ABAP CDS Views ile Performanslı Veri Modelleme — İleri Seviye Mimari Rehberi (advanced CDS modeling) apply directly—particularly parameterized CDS views for time-range filtering.

Your monitoring view should expose at minimum:

• All orders with confirmations pending beyond your threshold (e.g., operations completed in MES but not yet confirmed in SAP)

• Failed messages with their error details and retry count

• Quality lots awaiting usage decision past SLA

Common Failure Modes and How to Avoid Them

Let me share the failure modes I’ve seen most often, so you don’t have to learn them the hard way:

1. Order Status Not Checked Before Confirmation

Your MES will sometimes try to confirm an operation on an order that’s been technically completed, locked, or deleted in SAP. Always check order status (via STATUS_CHECK function module or JEST/TJ02T tables) before attempting a confirmation. A status check that takes 50ms saves you a failed transaction and a manual correction job.

2. Timezone Mismatches

MES systems often store timestamps in UTC. SAP production confirmations care about plant-local time for scheduling calculations. A timezone offset error can make your confirmation appear to occur before the order was released—which SAP will reject. Always align on timezone handling during design, not during go-live.

3. Missing Backflushing Logic

If your routing operations use backflushing for component consumption, SAP will automatically post goods issues when the operation is confirmed. Your integration needs to account for this—don’t double-post goods movements that backflushing will handle automatically. Check the backflushing indicator on the operation (AFVC-MGVRG) before posting separate goods movements.

4. No Idempotency Handling

Networks fail. Message brokers retry. Your confirmation BAPI call might get executed twice for the same MES event. Design your integration to handle duplicate messages gracefully—log the MES transaction ID and check for duplicates before calling the BAPI. This is basic, but surprisingly often missed.

The principles around clean error handling that I’ve referenced in SAP ABAP Performance Optimization apply to integration code as much as to any other ABAP program—maybe more so, because the error paths are more complex and the consequences of getting them wrong are immediate and visible on the production floor.

IoT Layer: Adding Machine Signals to the Picture

The frontier of MES-SAP integration is incorporating machine signal data directly. With OPC-UA becoming the standard industrial communication protocol, you can capture cycle times, temperatures, pressures, and machine states without operator input.

The architectural principle here is edge intelligence: don’t send every raw machine signal to SAP. Process signals at the edge or in the MES to generate business-relevant events. SAP should receive “Cycle completed—30.2 seconds, within tolerance” not a stream of raw sensor readings.

When a process parameter exceeds its tolerance, the event triggers three things simultaneously:

• A quality notification in SAP QM (BAPI_QUALNOT_CREATE)

• An alert in the MES for the operator

• An entry in your integration monitoring log

This closed-loop quality approach—where the machine itself triggers a quality response in the ERP—is what Industrie 4.0 looks like in practice. Not just connected machines, but machines that drive business processes.

Key Takeaways

• Choose your integration pattern deliberately: direct RFC for simple scenarios, message-based for resilience at scale, IoT-first for maximum real-time capability.

• The BAPI layer works well, but you must handle commits, rollbacks, and status pre-checks explicitly—SAP won’t protect you from a badly formed request.

• Quality integration is as important as production confirmation—design the usage decision workflow early.

• Build monitoring into your architecture from day one, not as an afterthought.

• Idempotency, timezone alignment, and backflushing awareness are the details that separate implementations that work from implementations that work in theory.

What’s Your Biggest MES Integration Challenge?

Every manufacturing environment has its own quirks—custom machine protocols, legacy MES systems that predate modern APIs, organizational dynamics between IT and operations teams. The architecture patterns here give you a foundation, but the real work is always in the details of your specific context.

I’d genuinely like to hear what you’re working through. Drop a comment below describing your integration scenario—whether you’re evaluating architectures for a new project or debugging an existing one—and let’s work through it together. And if this article gave you a useful framework, share it with a colleague who’s dealing with the same challenges. The shop floor deserves better than manual reconciliation and end-of-shift batch uploads.

Technical Leadership in SAP Projects: How Senior Architects Make Better Decisions Under Pressure
Every seasoned SAP architect has been there. It’s 11 PM, production is down, the business is losing thousands of dollars per hour, and three junior developers are staring at you waiting for a clear direction. In that moment, your technical depth matters—but it’s your technical leadership that determines the outcome. After years of navigating these high-stakes moments in SAP environments, I want to share what actually works when the pressure is on, and more importantly, how to build the decision-making muscle before you ever need it.

This article is for architects and senior developers who are either stepping into leadership roles or looking to sharpen their judgment in complex SAP project environments. We’ll cover structured decision frameworks, trade-off analysis, how to communicate architectural decisions effectively, and how to grow the next generation of SAP engineers around you.

Why Technical Leadership in SAP Is Uniquely Challenging

SAP landscapes are not typical software environments. You’re dealing with decades of business logic baked into customizations, strict upgrade cycles, deep integration dependencies, and organizations that are often risk-averse by nature. When you layer modern technologies—SAP BTP, ABAP RESTful Application Programming Model (RAP), CDS Views, event-driven architecture—on top of a legacy core, the architectural decisions become exponentially more complex.

Here’s what makes technical leadership in SAP particularly demanding:

• Legacy debt is invisible until it bites you. Years of undocumented enhancements, Z-programs, and BADI implementations create a minefield that only becomes visible mid-project.

• Business stakeholders speak a different language. Translating “we need to refactor the IDoc processing layer” into business risk and timeline impact is genuinely hard.

• The SAP ecosystem evolves faster than most teams can absorb. Clean Core, BTP extensions, S/4HANA migration—the rate of change demands continuous architectural re-evaluation.

• Decisions have long tails. A poor integration design choice made today might surface as a performance crisis two years into production. The architect who made the call may no longer even be on the project.

Understanding these constraints isn’t pessimism—it’s the foundation of good leadership. You make better decisions when you honestly acknowledge the environment you’re operating in.

The Architecture Decision Record (ADR): Your Most Underused Tool

One of the most impactful habits I’ve developed over the years is consistently writing Architecture Decision Records. An ADR is a short, structured document that captures why a significant technical decision was made, not just what was decided.

In SAP projects, this is particularly valuable. When a team three years from now is wondering why the system uses IDocs for vendor invoice integration instead of REST services, the ADR tells the story—including the constraints, the alternatives considered, and the trade-offs accepted.

ADR Template for SAP Projects

## ADR-0042: Use SAP Event Mesh for Order-to-Cash Integration

**Date:** 2025-01-15  
**Status:** Accepted  
**Deciders:** [Lead Architect, Integration Team Lead, BTP Platform Owner]

### Context
The Order-to-Cash process requires near-real-time event propagation 
between S/4HANA and three downstream systems (CRM, WMS, external 3PL).
Current IDoc-based approach introduces 15-30 minute delays and brittle 
point-to-point coupling.

### Decision
We will adopt SAP BTP Event Mesh as the central event broker for all 
Order-to-Cash domain events, replacing current IDoc RFC calls.

### Alternatives Considered
1. **Direct REST APIs** – Synchronous; creates tight coupling, 
   cascading failure risk under load.
2. **Enhanced IDocs with immediate delivery** – Reduces delay but 
   doesn't solve the coupling problem; limits future extensibility.
3. **SAP Integration Suite with polling** – Introduces unnecessary 
   complexity; polling defeats event-driven purpose.

### Consequences
- Positive: Loose coupling, independent scalability, audit trail via 
  event log, supports future consumer onboarding without sender changes.
- Negative: Team upskilling required on event-driven patterns; 
  added BTP licensing cost; eventual consistency model requires 
  business process alignment.

### Review Date: 2026-01-15

Notice the “Review Date” field. Good architectural decisions have expiry dates. What’s correct today may be wrong tomorrow, and building in a review cycle signals intellectual honesty to your team.

If you’re exploring event-driven architectures in SAP, our article on SAP BTP Event Mesh and building resilient integrations provides deep technical context that pairs well with the ADR approach described here.

Decision-Making Under Pressure: A Framework That Actually Works

When you’re in a production crisis or a contentious architecture review, structured thinking is your best ally. I use a lightweight framework I call SCOPE for time-pressured decisions:

• S – Situation clarity: What exactly is the problem? Don’t skip this. In SAP incidents, it’s common to spend 40 minutes solving the wrong problem.

• C – Constraints visible: What can’t we change? Business hours, system availability windows, release cycles, compliance requirements.

• O – Options on the table: Generate at least three options. The worst decisions come from false binary choices.

• P – Probability and impact: Quick mental model—what’s the likelihood each option creates new problems, and how severe?

• E – Execute and document: Make the call, own it, and write down why. Even a Slack message to the team counts.

The SCOPE framework won’t eliminate bad calls—nothing will. But it dramatically reduces the type of bad call where you react emotionally, skip analysis, or let the loudest voice in the room drive the decision.

Practical Example: Handling an ABAP Performance Crisis

Imagine a critical batch job in SD is timing out after the S/4HANA migration. Business is escalating. Here’s SCOPE in action:

• S: Batch job SELECT on VBAP and VBAK with joins runs 40+ minutes in S/4HANA vs. 8 minutes in ECC. Migration was two weeks ago.

• C: Can’t modify table structures; can’t postpone month-end close; only four-hour maintenance window available tomorrow morning.

• O: Option 1—rewrite SELECT to use CDS View with proper indexing. Option 2—introduce background parallelization using CALL FUNCTION IN BACKGROUND TASK. Option 3—apply SAP Note for known HANA optimizer issue with this query pattern.

• P: Option 3 is lowest risk if a relevant SAP Note exists (check first). Option 1 is medium risk—requires testing. Option 2 is highest risk—architecture change under time pressure.

• E: Check SAP Notes immediately. If no note, proceed with Option 1 in the maintenance window with a clear rollback plan.

For deeper context on ABAP performance investigation techniques, I recommend reading our ABAP performance optimization guide covering real-world bottleneck identification—the query analysis patterns there apply directly to scenarios like this.

Communicating Architecture to Non-Technical Stakeholders

One of the skills that separates good technical leads from great ones is the ability to make complex architectural concepts legible to business stakeholders—without dumbing them down or being condescending.

Here’s a technique I’ve refined over many project steering committees: the Three-Layer Communication Model.

Layer 1: Business Impact (30 seconds)

Lead with what changes for the business. “This change means your finance team will see vendor payment statuses update in near-real time instead of with a 30-minute delay.”

Layer 2: Why It Works (60 seconds)

Explain the mechanism without jargon. “Currently, our systems use a messaging format called IDoc that requires a scheduler to run every 30 minutes. We’re replacing it with an event-driven approach—think of it like switching from scheduled email digests to instant push notifications.”

Layer 3: Risk and Cost (as long as needed)

Be honest about trade-offs. This is where architects lose credibility by overselling. Acknowledge the upskilling cost, the testing effort, and the rollback complexity. Stakeholders respect honesty far more than false confidence.

The moment you start using acronyms in a business meeting without explaining them first, you’ve lost the room. Every IDoc, BAPI, RFC, and CDS acronym needs to earn its place in the conversation.

Growing the Next Generation: Mentorship as a Leadership Responsibility

Technical leadership isn’t just about making the right calls yourself—it’s about building a team that makes good calls when you’re not in the room. This is the hardest part of the job, and the most rewarding.

The Guided Discovery Method

When a junior developer brings you a problem, resist the urge to immediately give the answer. Instead, use what I call Guided Discovery:

• Ask what they’ve already tried. This surfaces their reasoning process, not just their conclusion.

• Ask what they think the root cause is. Often they’re closer to the answer than they realize.

• Offer a directional hint, not the solution. “Have you looked at how the selection conditions interact with the buffering strategy here?”

• Let them implement and debrief together. The learning sticks when they own the fix.

Yes, this takes longer than just fixing the bug yourself. But you’re not optimizing for today’s ticket—you’re building tomorrow’s senior developer.

Code Reviews as Teaching Moments

Your code reviews are one of your highest-leverage teaching tools. A few principles that have served me well:

• Comment on the code, never the person. “This approach may cause a performance issue under high load” vs. “You wrote this wrong.”

• Explain the why behind every critique. If you can’t explain why something should change, question whether it actually needs to.

• Acknowledge good work explicitly. Positive reinforcement in code reviews is criminally underused. If someone wrote a clean ABAP OOP implementation, say so.

• Use questions, not directives. “What happens if this internal table is empty when we reach this loop?” teaches better than “Add a CHECK statement here.”

Clean code practices in ABAP are a great foundation for these review conversations. Our guide on ABAP Clean Code and writing maintainable SAP code provides concrete standards that you can reference during reviews.

Trade-Off Analysis: The Core Skill That Separates Architects from Developers

Anyone can learn ABAP syntax. Anyone can configure a BTP service. The skill that genuinely differentiates a senior architect is the ability to reason clearly about trade-offs—and to make that reasoning transparent to the team.

A simple but powerful exercise: for every major technical decision, explicitly map it against four dimensions:

• Maintainability: How hard will this be to change in 2 years?

• Performance: What’s the behavior under peak load?

• Extensibility: Can we add new requirements without structural changes?

• Team capability: Does the team have the skills to build and operate this? If not, what’s the upskilling cost?

When you make these four axes explicit, decisions that seemed purely technical become organizational conversations. A microservices approach on BTP might score high on extensibility but low on team capability right now—and that’s a legitimate reason to choose a simpler architecture today with a clear migration path for later.

This kind of trade-off thinking is exactly what informs good decisions around process automation in SAP. The architectural considerations around RPA and AI integration—covered in our SAP BTP intelligent process automation article—are a practical application of this four-dimension framework.

Building Your Personal Decision-Making Track Record

Here’s a practice almost no one does but everyone should: keep a personal architecture journal. Not a formal document—a running log of decisions you made, why you made them, and what happened.

Every six months, review it. You’ll start to see patterns in where your instincts are strong and where they consistently lead you astray. This self-awareness is the foundation of genuine technical wisdom—the kind that can’t be taught in a course or read in a book.

Some questions worth journaling after major decisions:

• What information did I have, and what was I missing?

• Which constraints did I underestimate?

• Whose input did I not seek that I should have?

• If I made this call again tomorrow, would I change anything?

The architects I respect most are not the ones who never make mistakes. They’re the ones who learn from each decision at a structural level—and who actively look for their own blind spots.

Key Takeaways

• Use Architecture Decision Records to capture the why behind technical choices—not just the what.

• Apply a structured framework (like SCOPE) when making high-pressure decisions to avoid reactive, emotional choices.

• Communicate architecture in three layers: business impact first, mechanism second, risks and costs third.

• Invest in mentorship through Guided Discovery and high-quality code reviews—this is how you multiply your architectural impact.

• Make trade-off reasoning explicit across maintainability, performance, extensibility, and team capability.

• Keep a personal decision journal and review it regularly to identify and address your own blind spots.

What’s Your Experience?

Technical leadership in SAP is a craft that takes years to develop—and it’s still evolving. I’d genuinely like to hear from you: what’s the hardest architectural decision you’ve had to make under pressure in an SAP project, and what did you learn from it? Drop your thoughts in the comments below, or reach out directly. The conversations that happen in the comments section are often as valuable as the article itself.

If this resonated with you, share it with a colleague who’s stepping into an architect role—it might save them from a few painful lessons I had to learn the hard way.

I built a production AI agent on AWS. Not a demo, not a proof of concept — a real system with persistent memory, guardrails, CI/CD pipelines, and users who depend on it not going down at 2am.

The thing nobody tells you: the hard part isn’t the AI. The hard part is the infrastructure around it.

This series is my attempt to document everything I had to figure out the hard way — from architecture decisions in Part 1 all the way to cost optimisation in Part 6. The companion demo repo is at github.com/rajmurugan01/bedrock-agentcore-starter.

Let’s start at the beginning: why Amazon Bedrock AgentCore, and why not the “obvious” serverless approach.

The obvious approach: Lambda + Bedrock

If you’ve shipped anything serverless on AWS, your first instinct is Lambda. You know it, it has great tooling, CDK support is mature, and it scales to zero.

For a simple Bedrock wrapper — get a message, call InvokeModel, return a response — Lambda is fine. But the moment you add conversational state, it starts to crack.

Here’s what a real conversational AI agent needs:

1. Session state — the agent needs to remember what happened earlier in the conversation
2. Long-running processing — LLMs can take 30-90 seconds for complex multi-tool chains
3. Memory across sessions — the agent should know who the user is from previous conversations
4. Streaming responses — users expect tokens to appear progressively, not wait 60 seconds for a blob

Let’s look at how Lambda handles each of these.

Problem 1: Lambda’s 15-minute timeout

Lambda has a hard maximum execution timeout of 15 minutes. For a simple Q&A, that’s fine. But for an agentic loop — where the model calls tools, processes results, calls more tools, and reasons over everything — you can easily hit 5-10 minutes per complex interaction.

And I haven’t even mentioned the user’s session. If a user comes back after 20 minutes and continues the conversation, that’s a new Lambda invocation with zero context.

Problem 2: Session state storage

Lambda is stateless by design. Every invocation is independent. For conversational state, you need to:

1. Store session state somewhere (DynamoDB, ElastiCache, S3)
2. Load it at the start of every Lambda invocation
3. Save it at the end of every invocation
4. Handle the edge case where the Lambda times out mid-session
5. Build a session expiry and cleanup mechanism

That’s a lot of undifferentiated infrastructure for a problem that isn’t your core business.

Problem 3: Cross-session memory

Beyond session state, real assistants need memory — the ability to remember that a user’s preferred contact method is email, that they’re a premium customer, that they had a billing dispute last month.

With Lambda, you’d need to build this yourself: a vector database for semantic recall, a summarisation pipeline to consolidate old sessions, a retrieval step before each invocation. Entirely custom, entirely your problem to maintain.

What AgentCore actually does

Amazon Bedrock AgentCore is AWS’s managed infrastructure for running AI agents. It’s designed specifically for the workload pattern that Lambda handles poorly.

Here’s the mental model: AgentCore is a managed container orchestrator for long-running, stateful AI agent sessions. You ship a Docker container with your agent code. AgentCore handles:

• Container lifecycle — starts, stops, scales, and restarts containers
• Session routing — routes each user session to the right container instance
• Memory persistence — built-in Semantic, Summary, and UserPreference memory strategies
• JWT validation — validates Cognito (or custom) JWTs before your code even runs
• VPC networking — runs your containers inside your VPC without cold start penalties
• SSE streaming — handles the HTTP connection and SSE protocol for you

The architectural difference:

Lambda approach:
  User message → API Gateway → Lambda (cold start?) → load session from DynamoDB →
  call Bedrock → save session to DynamoDB → return response → Lambda exits

AgentCore approach:
  User message → AgentCore Runtime (JWT validated) → your container (already warm) →
  call Bedrock → response streams back → container stays warm for next message

The architecture we’re building

┌────────────────────────────────────────────────────────────────┐
│  GitHub Actions (OIDC)                                         │
│  ├── Build Docker (linux/amd64)                                │
│  ├── Push to ECR (:latest + :<sha>)                           │
│  └── update-agent-runtime CLI                                  │
└──────────────────────────────┬─────────────────────────────────┘
                               │
                    CDK v2 TypeScript deploys:
                               │
┌──────────────────────────────▼─────────────────────────────────┐
│  AWS Infrastructure (us-east-1)                                │
│                                                                │
│  AgentCore Runtime                                             │
│  ├── Cognito JWT authoriser                                    │
│  ├── AG-UI HTTP protocol (SSE streaming)                      │
│  └── Container: Python agent on port 8080                     │
│                                                                │
│  AgentCore Memory (3 strategies)                               │
│  Bedrock Guardrail (prompt injection + PII)                   │
│  CloudWatch alarms (token count + latency)                    │
└────────────────────────────────────────────────────────────────┘

Primary model: Claude Sonnet 4.6 with prompt caching
Background model: Amazon Nova Pro (cheap classification/summarisation)
CI/CD: GitHub Actions OIDC — no stored AWS credentials

Series roadmap

Full demo repo: github.com/rajmurugan01/bedrock-agentcore-starter

Originally published at rajmurugan.com. This is Part 1 of the Ultimate Guide to Building AI Agents on AWS with Bedrock AgentCore series.

I can now replay any AI agent stream from production.

Here’s how.
In my last post, I wrote about the four SSE bugs that break AI agent UIs at 2am — chunk boundary splits, missing token batching, hanging done states, and retry logic that retries the wrong things.

There’s a fifth problem I didn’t cover, because the fix didn’t exist yet.

What do you do the morning after something broke?

The stream is gone. The event sequence that caused the bug evaporated the moment the connection closed. You have a user complaint, maybe a generic error log, and zero ability to reproduce the issue locally because local dev doesn’t have real network conditions, real token rates, or the specific sequence of tool calls that triggered the failure.

Today I shipped AgentStreamRecorder to the agent-stream library to solve exactly this.

The debugging gap nobody talks about
When a REST API fails, you have the request and response in your logs. You can replay it with curl. You can write a regression test. The failure is reproducible.

When an AI agent stream fails mid-flight, you have nothing. The SSE connection is stateful and ephemeral. The events exist in a buffer, get consumed by the client, and are gone. If the frontend shows wrong state — tools that didn’t clear, progress that froze at 60%, text that truncated — you’re debugging from memory and screenshots.

We hit this at Praxiom repeatedly while building 36 production agent tools. The failure pattern was always the same:

1. User reports the stream “felt wrong” or the UI got stuck
2. You try to reproduce — works fine in dev
3. You add more logging — doesn’t help, the issue is in the event sequence, not individual events
4. You fix something and hope — no regression test possible
5. Issue reappears three weeks later in a different context
6. The root problem: we had full observability on every other layer of the stack except the stream itself.

What AgentStreamRecorder does
It’s a drop-in async wrapper. You add two lines to your existing endpoint and every stream gets saved to a .jsonl file automatically.

Before:

@app.post("/chat")
async def chat(req: ChatRequest):
    return agent_stream_response(run_agent(req.message))

After:

from agent_stream.recorder import AgentStreamRecorder

recorder = AgentStreamRecorder("streams/production.jsonl")

@app.post("/chat")
async def chat(req: ChatRequest):
    async def generate():
        async for sse_str in recorder.record(run_agent(req.message)):
            yield sse_str  # passes through unchanged

    return agent_stream_response(generate())

That’s it. The recorder wraps the async generator, tees each SSE event to the file, and re-yields the string unchanged. Your StreamingResponse sees nothing different. The client sees nothing different. Zero impact on latency.

What gets recorded
Each session in the .jsonl file starts with a header line, then one line per event:

{% embed {"session": "f3a2c1b0-...", "started_at": "2026-03-31T02:14:00+00:00", "t": 0}
{"t": 0.0,   "event": "token",      "data": {"text": "Hello, here"}}
{"t": 0.041, "event": "token",      "data": {"text": " is what I found"}}
{"t": 0.052, "event": "tool_use",   "data": {"tool_name": "web_search", "status": "running"}}
{"t": 0.894, "event": "tool_result","data": {"tool_name": "web_search", "duration_ms": 842, "status": "done"}}
{"t": 1.204, "event": "done",       "data": {"num_turns": 1, "tool_count": 1, "duration_ms": 1204}} %}

The t field is seconds since stream start — relative, not absolute — so files are portable across machines and time zones. The format is append-safe: each new session appends to the file with its own UUID and t=0 baseline, so you can keep one file per day and scan it with grep.

Three design decisions worth explaining:

.jsonl not binary. You can grep it. grep ‘”event”: “error”‘ production.jsonl instantly shows every stream that hit an error event, with timing. Binary formats are faster to write but terrible to investigate.

Relative timestamps. Absolute wall-clock timestamps tell you when something happened. Relative timestamps tell you how long it took. Relative is almost always what you need for debugging — “tool_result came back 842ms after tool_use” is more useful than two UTC timestamps you have to subtract.

Flush after every write. The file is flushed after every line, not buffered. If the process crashes mid-stream, you still have everything up to the last event. This matters — crashes are exactly when you most need the recording.

Replaying it
The agent-stream CLI (installed with the package) plays back recordings:

What sessions are in this file?

agent-stream replay production.jsonl –list

SESSION

The output is valid SSE piped to stdout. You can pipe it to a local dev server, feed it into a test harness, or just watch it in the terminal to see exactly what the client received.

The use case that made this necessary
Here’s the specific failure pattern that pushed us to build this. We had a multi-turn agent that called several tools in sequence. In production with certain inputs, the activeTools array in the React hook wasn’t clearing properly — a tool would finish but its name would stay in the “currently running” UI indefinitely.

Couldn’t reproduce it locally. The tool sequence was always different. Adding console logs to the hook showed correct state at each individual event, but the sequence mattered.

With AgentStreamRecorder, we recorded the failing session from production, replayed it through a local frontend, and watched the hook state update in real-time. Spotted the issue immediately: a tool_result event was arriving before the matching tool_use in one specific sequence, because two tools were running in parallel and the faster one resolved first. The hook was looking for tool names in order; the order wasn’t guaranteed.

Fifteen-minute fix. Would have taken days without the recording.

The bigger pattern: AI agents need a different kind of observability
Standard application observability — request logs, error rates, latency percentiles — doesn’t map cleanly onto AI agent streams. The unit of interest isn’t a request, it’s a session. The signal isn’t an error code, it’s an event sequence. The failure mode isn’t a stack trace, it’s a state machine reaching an unexpected state.

AgentStreamRecorder is a small step toward stream-native observability. Each .jsonl session is a complete, replayable trace of exactly what the agent did and how long each step took. You can diff two sessions to understand why one succeeded and one failed. You can build a test suite from real production recordings. You can grep across thousands of sessions to understand behavioral patterns at scale.

None of this is revolutionary. It’s the kind of thing you’d take for granted in any mature system — request tracing, structured logging, replay-from-log. We just haven’t had it for agent streams.

Getting it
pip install agent-event-stream
from agent_stream.recorder import AgentStreamRecorder

Full source, spec, and React client hook at github.com/abhichat85/agent-stream.

Extracted from Praxiom — Product Intelligence that compounds. Think Cursor for Product Managers.

BeyondTrust’s Phantom Labs just published a report on a command injection vulnerability in OpenAI’s Codex. It’s patched now, but the attack pattern matters because it’s exactly the kind of thing vibe coders won’t see coming.

What Happened

Codex runs tasks inside managed containers that clone your GitHub repo and authenticate using short-lived OAuth tokens. The vulnerability: branch names weren’t sanitized before being passed to shell commands during environment setup.

An attacker could craft a malicious branch name that injects arbitrary shell commands. Those commands execute inside the container with access to your GitHub token.

The attack worked across:

• The Codex web interface
• The CLI
• The SDK
• IDE integrations

Worse: it could be scaled. Embed a malicious payload in a branch name, and every developer who interacts with that repo through Codex gets compromised.

What Could Be Stolen

The GitHub OAuth tokens Codex uses aren’t just read tokens. In enterprise environments where Codex has broad permissions:

• Full read/write access to repositories
• Workflow trigger permissions (CI/CD pipelines)
• Organization-level access depending on token scope

One compromised branch name. Every Codex user on the repo exposed.

Why This Matters for Vibe Coders

This vulnerability was found by professional security researchers at BeyondTrust. Most vibe coders:

1. Don’t review branch names for injection payloads
2. Don’t audit what permissions their AI coding tools have
3. Don’t know what an OAuth token scope even is
4. Trust that “it’s a managed container” means it’s safe

The attack surface isn’t your code. It’s your tools.

The Bigger Picture

This dropped the same day Claude Code launched Computer Use (mouse and keyboard control). Two separate stories, same lesson:

AI coding agents are live execution environments with access to your credentials.

They’re not just autocomplete. They run commands, clone repos, access tokens, and now control your screen. Every new capability is a new attack surface.

In the last 7 days:

• LiteLLM supply chain attack hit 95M monthly downloads (TeamPCP campaign)
• Same attacker compromised Trivy (vulnerability scanner) and KICS (IaC analyzer)
• OpenAI Codex command injection exposed GitHub tokens
• Claude Code gained mouse and keyboard access

The tools we trust to write and test our code are becoming the primary attack vector.

What To Do

1. Audit your AI tool permissions. What repos can Codex access? What scope do the tokens have? Minimize to read-only where possible.
2. Pin your dependencies. TeamPCP compromised packages that millions install without version pinning.
3. Don’t trust container isolation alone. The Codex containers had network access. “Managed” doesn’t mean “secure.”
4. Scan your deployed apps. If you built it with AI tools, scan it before users find what you missed. VibeCheck is free.
5. Check for exposed secrets. Branch names, commit messages, config files. AI tools don’t flag these by default.

OpenAI patched this one. The next vulnerability in the next AI coding tool hasn’t been found yet.

Building VibeCheck, a free security scanner for vibe-coded apps. Follow @solobillionsHQ for daily vibe coding security updates.

There is a small board on a shelf that never sleeps.

No fan noise. No screen. Just a faint warmth if you press a finger against it long enough. The LED stopped blinking weeks ago. Or maybe it still is, and you stopped noticing.

It’s still running.

And that’s where people get it wrong.

They think “running” means stable. Predictable.
Done.

It doesn’t.

It Doesn’t Stay the Same Device

The first version of your ESP32 is the one you flashed. Clean. Intentional. Every line of code accounted for.

That version doesn’t last.

Over time, the device drifts. Not in some mystical sense. In a very literal accumulation of state. Buffers fill and empty. Memory fragments. WiFi reconnects hundreds of times. Timers tick past thresholds you never tested. Edge cases stop being edge cases because given enough hours, everything happens.

You wrote logic assuming a day. Maybe a week.
Leave it for a month and you’re no longer running your code. You’re running everything your code didn’t anticipate.

There’s a difference.

Memory Starts to Behave Like a Surface, Not a Container

On paper, the ESP32 has enough RAM for most lightweight systems. In practice, that RAM becomes uneven over time.

Heap fragmentation is the quiet one. It doesn’t crash immediately. It just reshapes what’s available. Allocations that worked on day one start failing in strange ways. Not clean failures either. Partial allocations. Corrupted data.

Functions returning values that feel slightly off.
You restart the device and everything looks fine again.

That should bother you.

Because it means the system only works in a freshly initialized state. Which means it’s not stable. It’s just temporarily aligned.

Long-running devices expose this quickly.

Especially if you’re doing dynamic allocation, string operations, or anything involving WiFi stacks and JSON parsing.

It’s not dramatic. It’s worse than that. It’s subtle.

WiFi Becomes a Personality Problem

WiFi on the ESP32 is good. Until it isn’t.

When you leave a device running continuously, you start to see patterns that don’t show up in short sessions. Networks drop for a few seconds. Routers rotate channels. DHCP leases expire. Signal strength shifts based on time of day, interference, other devices.

Your ESP32 has to live through all of that.

And most sketches don’t handle it properly.

They check WiFi.status() and reconnect. That’s not enough. Sometimes the stack thinks it’s connected when it isn’t. Sometimes reconnect loops lock up other tasks. Sometimes the device enters a state where it’s technically online but functionally dead. No data moves.

You won’t notice unless you’re logging externally.
From the outside, it looks fine. The LED is still on.

Inside, it’s stalled.

Heat Is Low, But Not Zero

People treat the ESP32 like it’s immune to heat because it runs cool.

That’s only half true.

Yes, it won’t burn your fingers. But over long durations, even small thermal fluctuations matter. Especially if the board is enclosed, or sitting near other electronics, or powered through questionable regulators.

Temperature affects timing. Stability. RF performance.

Not enough to break things instantly. Enough to shift behavior over time.

You start seeing weird resets. Slight timing drift. Sensor readings that slowly diverge.

Nothing obvious. Just enough to make you doubt your data.

Power Quality Starts to Matter More Than Code

For short runs, almost any USB cable and power source will do.

For 24/7 operation, power becomes the foundation.
Cheap adapters introduce ripple. Long cables drop voltage. Sudden load spikes from WiFi transmissions can dip the supply just enough to trigger brownouts.

The ESP32 has brownout detection, but many people disable it when it becomes annoying during development.

That decision comes back later.

Now instead of clean resets, you get undefined behavior. Partial execution. Corrupted state.
A device that “runs fine” for hours suddenly locks up after three days.

You blame the code. It might not be the code.

Time Stops Being Abstract

Most hobby projects ignore time beyond millis().
Leave something running for weeks and time becomes a real dependency.

Timers overflow. Scheduling drifts. Events that rely on precise intervals begin to slide. If you’re syncing with external systems, even small inaccuracies compound.

Without NTP or some form of correction, your device’s sense of time slowly diverges from reality.

At first it’s seconds. Then minutes.

Then your logs stop lining up with anything else.
If your system depends on ordering events, that’s not a minor issue.

Logging Becomes the Only Truth

When a device runs continuously, you stop being present for most of its behavior.

That means your understanding is only as good as your logs.

And most ESP32 setups barely log anything.

A few serial prints during development. Maybe some debug output. Then it gets removed to “clean things up.”

Now something goes wrong at hour 72 and you have no record of what led to it.

You’re blind.

Real 24/7 systems treat logging as a core feature, not an afterthought. Even if it’s just rotating logs on an SD card or pushing minimal telemetry to a server.

Without that, you’re guessing.

And guessing feels a lot like debugging until it doesn’t.

The Device Starts Interacting With the Real World

This is the part people underestimate.

A continuously running ESP32 stops being a test object and becomes part of an environment.

It interacts with networks that change. Power that fluctuates. Inputs that aren’t controlled. People who don’t know it exists.

Maybe it’s scanning RFID tags. Maybe it’s reading sensors. Maybe it’s acting as a node in something larger.

Over time, it accumulates context.

Patterns emerge. Not in a clean dataset sense. In a messy, lived-in way. Missed reads. Duplicate events. Noise that wasn’t present during testing.
You start realizing your logic was built for a lab, not for reality.

And reality is not consistent.

Stability Is Engineered, Not Implied

Leaving an ESP32 running 24/7 doesn’t automatically make it stable.

It reveals whether it is.

Most setups fail this test in quiet ways. Not explosions. Not obvious crashes. Just gradual degradation. A system that becomes less correct over time.

To counter that, you have to design for longevity from the start.

Not complicated changes. Just intentional ones:

• Watchdogs that actually reset the system when it stalls
• Periodic soft reboots to clear state
• Defensive memory handling instead of optimistic allocation
• Real reconnection logic for WiFi, not just retries
• External logging so you can see what happened when you weren’t there

None of this is glamorous.
But it’s the difference between a project and a system.

It Becomes Something You Don’t Fully Control

There’s a point where a long-running device stops feeling like a tool and starts feeling like a process.

It runs whether you check it or not. It fails in ways you didn’t directly cause. It recovers sometimes. Other times it doesn’t.

You can tighten it. Monitor it. Reset it.

But you don’t sit inside every execution path anymore.

That distance matters.

Because it forces you to think differently. Less about writing code that works now. More about writing code that survives being left alone.

The Quiet Shift

If you’ve only ever powered your ESP32 during active sessions, you’re seeing a curated version of it.

Clean boots. Short timelines. Controlled inputs.
Leave it running for weeks and you get the unfiltered version.

Not broken. Just honest.

It shows you where your assumptions were thin. Where your system depends on luck. Where stability was implied instead of built.

Most people don’t run their devices long enough to see this.

Which is why their projects feel solid.
Until they aren’t.

Feel like supporting the publication? Look no further than my curated guides on cybersecurity, diy electronics, and more on my gumroad. The guides below pair very well with the content of this post, and these guides are currently keeping me “fed ‘n’ bed” as I am currently in a unique and challenging situation in my life. Any support means the world, and includes lifetime updates and access to my personal email for debugging/discourse. Thanks again, if you read this far- you are obviously on the same wavelength. Keep pushing. 

The BadUSB Factory: Convert Any Flash Drive Into a Weapon

The Ultimate Arduino Project Compendium