Mert

TL;DR

Deploy a CloudFormation stack, configure ONTAP audit logging, and see structured file access events in Datadog Log Explorer within minutes — no EC2, no NFS mounts, no agents. This post walks through the full implementation: CloudFormation template, Lambda handler code, Datadog field mapping, and operational validation.

What We’re Building

In Part 1, I introduced the architecture: FSx for ONTAP audit volume → S3 Access Point → EventBridge Scheduler → Lambda → Datadog. Now let’s build it.

By the end of this post, you’ll have:

• A deployed CloudFormation stack with Lambda, Scheduler, DLQ, and alarms
• ONTAP audit events flowing into Datadog Log Explorer
• Structured attributes (@attributes.svm, @attributes.user, @attributes.operation, @attributes.path, @attributes.client_ip, @attributes.result) ready for search, filtering, and Datadog facet creation
• An operational CloudWatch dashboard monitoring pipeline health

Prerequisites

Before deploying, you need:

1. FSx for ONTAP file system with an SVM configured for audit logging
2. FSx for ONTAP S3 Access Point attached to the audit volume
3. Datadog account (free trial works) with an API Key
4. API Key in Secrets Manager:

aws secretsmanager create-secret 
  --name fsxn-datadog-api-key 
  --secret-string '{"api_key":"<your-dd-api-key>"}' 
  --region ap-northeast-1

1. ONTAP audit logging enabled:

# Time-based rotation for quick validation
vserver audit create -vserver <svm-name> -destination /audit_log 
  -events file-ops 
  -format evtx 
  -rotate-schedule-minute 0,5,10,15,20,25,30,35,40,45,50,55
vserver audit enable -vserver <svm-name>

For quick validation, use time-based rotation. If you only use -rotate-size, low-volume environments may not produce rotated audit files within the expected validation window. Adjust the -events list based on what you want to audit.

Important: Enabling vserver audit is only one part of file access auditing. Make sure the target SMB folders have SACLs configured, or NFSv4 ACL audit flags are set for NFS workloads. Otherwise, the audit pipeline may be healthy but no file access events will be generated.

For detailed ONTAP-side setup, including audit volume sizing, SACL/NFSv4 ACL examples, and source health checks, see the repository’s ONTAP Audit Setup Guide and Operational Guide.

1. Verify how audit files appear via S3 API (to set AuditLogPrefix correctly):

aws s3api list-objects-v2 
  --bucket <fsx-s3-access-point-arn-or-alias> 
  --max-keys 10 
  --region ap-northeast-1

Set AuditLogPrefix to match the key prefix you see. If the access point is attached directly to the audit volume root, this may be empty.

Note: /audit_log is the ONTAP namespace path. The S3 object key prefix can differ depending on the access point attachment, so always verify with list-objects-v2.

The CloudFormation Stack

The Datadog integration deploys as a single self-contained stack:

aws cloudformation deploy 
  --template-file integrations/datadog/template.yaml 
  --stack-name fsxn-datadog-integration 
  --parameter-overrides 
    FsxS3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap 
    DatadogApiKeySecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:fsxn-datadog-api-key 
    DatadogSite=ap1.datadoghq.com 
    AuditLogPrefix=<prefix-from-list-objects-v2> 
    ScheduleRate="rate(5 minutes)" 
  --capabilities CAPABILITY_NAMED_IAM 
  --region ap-northeast-1

What Gets Created

Key Parameters

The Lambda Handler

The handler follows a straightforward flow:

Scheduled invocation
  → List objects from FSx for ONTAP S3 AP (via S3 ListObjectsV2)
  → Filter by checkpoint (skip already-processed files)
  → For each new file:
      → Read via S3 GetObject
      → Detect format (EVTX magic bytes or XML declaration)
      → Parse into normalized events
      → Format for Datadog Logs API v2
      → Batch (≤5MB, ≤1000 items per request)
      → Ship with exponential backoff (max 3 attempts)
  → Update checkpoint

Datadog API Limits

The Datadog Logs API v2 enforces the following per-request limits (docs):

• Maximum payload size (uncompressed): 5MB
• Maximum size for a single log: 1MB (larger logs are truncated, not rejected)
• Maximum array size: 1000 entries

The shipper batches conservatively below these limits.

Core Shipping Logic

def _ship_to_datadog(logs: list[dict], api_key: str) -> int:
    """Ship normalized logs to Datadog Logs Intake API v2.

    If any batch fails after retries, raise an exception so the Lambda
    invocation is treated as failed and the checkpoint is not advanced.
    """
    shipped = 0
    failed_batches = 0

    for batch in _create_batches(logs):
        if _send_batch(batch, api_key):
            shipped += len(batch)
        else:
            failed_batches += 1

    if failed_batches:
        raise RuntimeError(f"{failed_batches} batch(es) failed after retries")

    return shipped

Checkpoint Semantics

The checkpoint is advanced only after all batches for an audit log file are successfully delivered to Datadog. If any batch fails after retries, the Lambda invocation fails (raises an exception) and the checkpoint is not updated.

This makes the pipeline at-least-once: the same audit file may be retried on the next scheduled invocation, so downstream queries should tolerate duplicate events. For production, consider adding a deterministic event ID derived from the audit file key and event record offset to support deduplication where your observability platform supports it.

Because EventBridge Scheduler invokes Lambda asynchronously, a failed invocation (unhandled exception) triggers Lambda’s built-in retry behavior (up to 2 retries by default). After all retries are exhausted, the event payload is sent to the configured DLQ.

Retry with Exponential Backoff

def _send_batch(batch: list[dict], api_key: str) -> bool:
    """Send a single batch with retry on 429/5xx, up to MAX_RETRIES attempts."""
    for attempt in range(MAX_RETRIES):
        response = http.request(
            "POST",
            DATADOG_LOGS_URL,
            body=json.dumps(batch).encode("utf-8"),
            headers={
                "Content-Type": "application/json",
                "DD-API-KEY": api_key,
            },
        )
        if response.status < 300:
            return True
        if response.status == 429 or response.status >= 500:
            time.sleep(2 ** attempt + random.uniform(0, 1))  # jitter
            continue
        # Client error (4xx) — don't retry
        return False
    return False

The implementation uses exponential backoff with jitter (2^attempt + random) to avoid synchronized retries when multiple Lambda invocations hit vendor-side throttling simultaneously. Note that MAX_RETRIES in the code represents the total number of attempts, not retries after an initial attempt.

API Key Caching

The API key is fetched from Secrets Manager once per Lambda execution context (cold start) and cached in a module-level variable. This avoids per-invocation Secrets Manager calls:

_api_key_cache: str | None = None

def get_api_key() -> str:
    global _api_key_cache
    if _api_key_cache:
        return _api_key_cache
    response = secrets_client.get_secret_value(SecretId=API_KEY_SECRET_ARN)
    secret = json.loads(response["SecretString"])
    _api_key_cache = secret.get("api_key", secret.get("dd_api_key", response["SecretString"]))
    return _api_key_cache

Datadog Field Mapping

Every audit event arrives in Datadog with structured attributes. The Lambda sends these via the Datadog Logs API v2 payload fields (ddsource, hostname, service, message) and custom attributes nested under attributes:

Set DatadogSite to your Datadog site, such as datadoghq.com (US1), datadoghq.eu (EU1), or ap1.datadoghq.com (AP1/Tokyo). The site determines the API endpoint.

For the full cross-vendor mapping (Datadog, Splunk, Elastic, OpenTelemetry), see the Normalized Event Schema.

Datadog Search Queries

# All FSx for ONTAP audit events
source:fsxn

# Failed access attempts
source:fsxn @attributes.result:Failure

# Specific user activity
source:fsxn @attributes.user:"[email protected]"

# Delete operations on sensitive paths
source:fsxn @attributes.operation:delete @attributes.path:"/vol/data/confidential/*"

# Pipeline processing metadata
source:fsxn @attributes._pipeline.source_file:*

In Part 3, we’ll turn these queries into Datadog Monitors for ARP ransomware detection and suspicious file activity alerting.

Investigation Query Starters

When investigating an incident, start with these patterns:

For high-volume environments, avoid grouping by full file path unless needed. Consider deriving a lower-cardinality field such as a path prefix or data area classification.

Operational Validation

Quick Validation (5–10 minutes)

With a 5-minute audit rotation and 5-minute Scheduler interval, the first events typically appear within a few minutes, but allow up to 10 minutes depending on timing.

Before waiting for logs, generate a test file operation on the audited SMB/NFS share — such as creating and deleting a small test file — to ensure ONTAP produces an audit event.

# 0. Get stack outputs (log group name, DLQ URL, etc.)
aws cloudformation describe-stacks 
  --stack-name fsxn-datadog-integration 
  --query 'Stacks[0].Outputs' 
  --region ap-northeast-1

# 1. Confirm Scheduler is invoking Lambda
aws logs filter-log-events 
  --log-group-name <LambdaLogGroupName from outputs> 
  --start-time $(python3 -c "import time; print(int((time.time()-300)*1000))") 
  --region ap-northeast-1

# 2. Confirm DLQ is empty
aws sqs get-queue-attributes 
  --queue-url <dlq-url> 
  --attribute-names All 
  --query 'Attributes.ApproximateNumberOfMessages'

# 3. Search in Datadog
#    source:fsxn

CloudWatch Dashboard

The stack includes a pre-built dashboard (fsxn-datadog-integration-health) with:

• Lambda Errors & Throttles
• Lambda Duration (avg/max)
• Lambda Invocations
• DLQ Depth

For production, consider publishing custom metrics such as files processed, events shipped, batch failures, and checkpoint lag to gain deeper pipeline observability beyond Lambda-level metrics.

What to Watch For

Troubleshooting

Old Timestamps May Not Appear in Log Explorer

The Datadog Logs API accepts log events with timestamps up to 18 hours in the past. If your audit files are rotated or processed too late, older events may not appear as expected in Log Explorer.

Fix: Use a time-based ONTAP audit rotation schedule and a Scheduler frequency that keeps processing well within the 18-hour window.

Gzip Compression Issue (AP1 Site)

During E2E validation, gzip-compressed payloads were accepted (HTTP 202) but not indexed on the AP1 site. The ENABLE_GZIP parameter defaults to false for this reason.

S3 Access Point Timeout in VPC

If Lambda is in a VPC with only an S3 Gateway Endpoint, reads from FSx for ONTAP S3 Access Points will timeout. Add NAT Gateway or deploy Lambda outside VPC.

Day-2 Operations

DLQ Replay

This stack uses an SQS queue as the Lambda asynchronous invocation DLQ. Because the DLQ is attached to Lambda (not an SQS source queue), sqs start-message-move-task cannot redrive messages automatically.

For replay, inspect the DLQ message, identify the failed invocation payload, and re-invoke Lambda manually:

# Inspect failed messages
aws sqs receive-message 
  --queue-url <dlq-url> 
  --max-number-of-messages 1 
  --attribute-names All 
  --message-attribute-names All

After fixing the root cause (e.g., expired API key, Datadog site misconfiguration), re-run the scheduled processor:

aws lambda invoke 
  --function-name <lambda-function-name> 
  --cli-binary-format raw-in-base64-out 
  --payload '{}' 
  --region ap-northeast-1 
  replay-output.json

In this pattern, replay usually means re-running the scheduled processor after fixing the root cause. Because the checkpoint is not advanced on failed delivery, the same audit file remains eligible for processing on the next invocation. This does not re-submit the DLQ message itself — it re-runs the processor so files whose checkpoints were not advanced can be picked up again.

For production, consider adding a dedicated replay Lambda that reads DLQ messages, validates the payload, and re-submits failed processing requests in a controlled way.

Checkpoint Reset (Reprocess All Files)

⚠️ Warning: Resetting the checkpoint causes previously processed audit files to be eligible for reprocessing. This can generate duplicate logs in Datadog. Use only for controlled replay or testing.

aws dynamodb delete-item 
  --table-name fsxn-observability-audit-checkpoint 
  --key '{"svm_name": {"S": "svm-prod-01"}, "file_key": {"S": "LATEST"}}'

Teardown

aws cloudformation delete-stack 
  --stack-name fsxn-datadog-integration 
  --region ap-northeast-1

Deleting the stack does not affect ONTAP audit logging or data on the FSx for ONTAP volume.

Cost Estimate

For a typical deployment (1 SVM, 100MB audit logs/day, 5-minute schedule):

Cost numbers are illustrative. Assume a 5-minute schedule, 5-second average runtime, and 100MB/day of audit logs. NAT Gateway pricing is regional and includes hourly charges plus per-GB data processing charges. Check the AWS Pricing Calculator for your target Region.

Important: Datadog ingest and retention costs are not included in this AWS-side estimate and can become the dominant cost driver for high-volume audit policies, especially when read auditing is enabled.

Evidence retention: This pipeline optimizes search and alerting via normalized events in Datadog. If you need audit evidence retention for compliance, design raw EVTX/XML retention separately on the audit volume or in an archive path.

Cost control: For high-volume environments, consider a tiered strategy: send security-relevant operations such as deletes, permission changes, and failed access to indexed logs; reduce, archive, or exclude noisy read events only if your audit and compliance requirements allow it.

Compare this to an always-on EC2 collector instance, plus EBS, patching labor, and agent licensing.

What’s Next

In Part 3, we’ll add event-driven security alerting:

• ONTAP Autonomous Ransomware Protection (ARP) detection
• EMS webhook → API Gateway → Lambda → Datadog
• Datadog Monitor configuration for instant alerts
• Incident response workflow

Datadog is the first E2E-verified integration in this pattern library; the same structure will be used for the remaining vendor integrations as they are validated.

Questions about the Datadog integration? Drop a comment below.

Previous: Part 1 — Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
Next: Part 3 — Event-Driven Ransomware Detection with ONTAP ARP + Datadog

Most people assume the difference between good thinking and bad thinking is intelligence.

It isn’t.

More often, it comes down to cost — how much mental effort it takes to arrive at a useful answer. Some answers require slow, deliberate reasoning. Others emerge almost instantly through pattern recognition or experience.

Human thinking isn’t a single process. It’s a collection of reusable patterns — some fast, some careful, some surprisingly fragile depending on context.

AI doesn’t replace these patterns. It changes what they cost to use.

1. Structured Thinking (Slow, Deliberate Reasoning)

This is the kind of thinking people usually mean when they talk about “deep work” or “careful reasoning”. It prioritises correctness over speed, and it tends to feel effortful.

Decomposition

Large problems rarely stay large once you start breaking them apart. Splitting something into smaller components makes it easier to understand, reason about, and eventually solve. In software, this is the difference between an overwhelming system and a set of manageable modules.

AI is especially good at this first pass. It can outline a system or break down a problem in seconds — something that would normally take a whiteboard session and some back-and-forth.

Abstraction

Abstraction is the act of ignoring detail on purpose. You don’t need to understand everything to use something effectively — just the parts that matter.

Good engineers instinctively group things that change together and separate things that don’t. That decision — what to include and what to ignore — is still very human. AI can assist with structure, but judgment still sits on your side.

Iteration

Most real progress doesn’t happen in one leap. It happens through repeated cycles: try something, observe what breaks, adjust, repeat.

What AI changes here is speed. Iteration cycles that once took hours or days can now happen in minutes. The loop stays the same — it just becomes much cheaper to run.

Hypothesis Testing

Good debugging isn’t guessing randomly. It’s forming small, testable ideas and checking them one at a time.

AI helps by suggesting likely causes or next steps, but the grounding still comes from reality — logs, outputs, behaviour. It can accelerate thinking, but it can’t replace validation.

Systems Thinking

At some point, you stop looking at parts and start looking at interactions. Changing one thing affects others in ways that aren’t always obvious — performance, reliability, consistency.

AI can map dependencies reasonably well, but real systems tend to reveal their complexity only when they’re running under pressure. That’s where intuition still matters.

Edge Cases

Most real bugs don’t come from the “normal path”. They appear at the edges — empty inputs, failed requests, race conditions, unexpected states.

AI can suggest some of these, but domain-specific edge cases usually come from experience rather than inference.

2. Fast Thinking (Heuristics and Shortcuts)

Fast thinking is what gets used most of the time in real life. It’s not careful — it’s efficient. It trades precision for speed.

Heuristics

Rules of thumb like “start simple”, “avoid premature optimisation”, or “readability over cleverness” exist because they work often enough to be useful.

AI tends to generate a default version of these heuristics, which is helpful — but also risky if you accept them without thinking. The baseline gets easier, but that also makes it easier to stop questioning it.

Satisficing

Most decisions aren’t about finding the best option. They’re about finding one that’s good enough to move forward.

AI lowers the threshold for “good enough” dramatically. That changes behaviour: people stop searching earlier because acceptable solutions appear faster.

Anchoring

The first answer you see tends to shape everything that follows.

With AI, that anchor is often the first generated response — fluent, confident, and structured. That makes it especially sticky, even when it’s not optimal.

Availability Bias

We tend to estimate likelihood based on what comes to mind easily.

AI shifts what becomes “easy to recall” toward patterns it has seen most often in its training data. That subtly biases thinking toward common solutions.

Narrative Thinking

Humans like stories. We compress complexity into something linear and understandable.

AI is extremely good at producing narratives, even when reality is messy. That makes explanations clearer — but also sometimes cleaner than reality actually is.

3. Simulation and Future Thinking

Mental Simulation

We constantly run mental “what if” scenarios before acting.

AI expands this space dramatically. Instead of imagining two or three outcomes, you can explore dozens in seconds.

Regret Minimisation

Many decisions are really about avoiding future regret.

When alternatives are easy to generate, comparing outcomes becomes more explicit — and sometimes more overwhelming.

Opportunity Cost

Every decision excludes others, but we don’t usually feel that cost clearly.

AI makes execution cheaper, which ironically makes decision-making more important. When doing is easy, choosing becomes the real bottleneck.

4. The Meta Layer — Thinking About Thinking

The hardest shift isn’t technical. It’s awareness.

It’s knowing when you’re reasoning carefully versus when you’re just following a fluent suggestion that feels like reasoning.

AI blurs that boundary. You can easily feel like you’ve thought something through when you’ve actually just selected from a well-phrased option.

That makes feedback loops more important than ever — checking, testing, and re-evaluating what you assume is true.

Closing Thought

Thinking isn’t a single skill. It’s a set of patterns we switch between constantly, usually without noticing.

Most errors don’t come from lack of intelligence. They come from using the wrong pattern for the situation.

AI doesn’t change that.

It just makes every pattern cheaper — and the switching between them less visible.

Which means the real skill is no longer just thinking well.

It’s noticing what kind of thinking you’re doing while it’s happening.

TL;DR

FSx for ONTAP file access audit logs are usually consumed through EC2-based patterns — mounted audit volumes and agent-based forwarders such as Splunk Universal Forwarder. This series explores an EC2-free alternative: configure ONTAP to write audit logs to an audit volume, expose that volume through an FSx for ONTAP S3 Access Point, use EventBridge Scheduler to invoke Lambda, and ship normalized events to observability platforms such as Datadog, Splunk, New Relic, Grafana Cloud, Elastic, and OpenTelemetry-compatible backends.

What This Post Covers

This post introduces the architecture and the open-source pattern library. It does not yet cover:

• Full Datadog deployment walkthrough (Part 2)
• Vendor-specific field mappings
• Cost/performance benchmarking
• ARP + EMS webhook + Datadog alerting (Part 3)
• FPolicy binary protocol internals (future post)

The Problem Nobody Talks About

You’re running Amazon FSx for NetApp ONTAP. You’ve enabled file access auditing because compliance requires it — or because you genuinely want to know who’s accessing what on your file shares.

But where do those audit logs go?

If you followed the official AWS blog post, you likely ended up with EC2-based collectors: syslog-ng for cluster/admin audit forwarding, and a mounted audit volume plus Splunk Universal Forwarder for file access audit logs. It works. But now you have:

• EC2 instances to patch and maintain
• NFS mounts to the audit volume
• syslog-ng configuration for admin audit forwarding
• Splunk Universal Forwarder configuration for file access logs
• A single point of failure unless you build your own HA pattern
• Vendor lock-in to Splunk’s agent-based model

What if you could replace that EC2-based collector pattern with managed services — Lambda reads audit logs via S3 APIs, no NFS mount required — and ship to any observability platform?

That’s the goal of this project.

Important Distinction: Two Types of ONTAP Audit

Before diving in, a clarification. FSx for ONTAP has two distinct audit mechanisms:

1. Cluster/admin activity audit logs — Administrative operations (CLI/API commands). These are forwarded via syslog to a log destination, as described in the AWS blog.

2. File access audit logs — SMB/NFS file operations (open, read, write, delete, permission changes). These are recorded based on ONTAP audit policies and SACLs/NFSv4 ACLs, stored in EVTX or XML format, depending on your ONTAP audit configuration, on an audit volume inside the SVM.

In this series, “audit logs” refers to file access audit logs (type 2). The cluster admin audit forwarding via syslog is a separate concern.

The EC2-Free Alternative

I’m building an open-source pattern library that targets 9 observability vendors using Lambda, EventBridge Scheduler, and ECS Fargate — eliminating the need for self-managed EC2 instances.

This is EC2-free, not necessarily Lambda-only:

• The audit-log and EMS paths are Lambda patterns (scheduled and event-driven respectively).
• The FPolicy path uses ECS Fargate because ONTAP FPolicy requires a persistent TCP listener.

How Audit Logs Flow

ONTAP’s file access auditing writes rotated audit log files to a configured destination path inside the SVM. In this project, that destination is an audit volume exposed through an FSx for ONTAP S3 Access Point. Lambda does not mount NFS or SMB; it reads the rotated audit log files through S3 APIs.

Because this pattern does not rely on S3 ObjectCreated events from FSx for ONTAP S3 Access Points, the audit processor is invoked on a schedule and uses checkpointing to process only newly rotated log files.

FSx for ONTAP audit configuration (`vserver audit`)
    │
    ▼ audit logs written to /audit volume
Audit volume exposed via FSx for ONTAP S3 Access Point
    │
    ▼ EventBridge Scheduler (periodic invocation)
Lambda audit processor (Python 3.12)
    │
    ▼ parse EVTX/XML → normalize → vendor API
Datadog / Splunk / New Relic / Grafana / Elastic / ...

The key shift from the EC2 pattern: Lambda does not mount the audit volume over NFS or SMB. It reads rotated ONTAP audit log files through an FSx for ONTAP S3 Access Point using S3 APIs, while the data itself remains on the FSx for ONTAP file system.

A note on FSx for ONTAP S3 Access Points: FSx for ONTAP S3 Access Points let applications use S3 APIs to access data that still resides on FSx for ONTAP volumes. They are excellent as a serverless access boundary, but they are not the same as standard S3 buckets. In particular, you should not rely on S3 ObjectCreated notifications from an FSx for ONTAP S3 Access Point. Instead, this project uses EventBridge Scheduler plus checkpointing to discover and process newly rotated audit log files.

Three Event Sources, One Architecture

FSx for ONTAP generates observability data through three distinct channels:

1. File Access Audit Logs (FSx for ONTAP S3 AP)

Depending on your ONTAP audit configuration and SACL/NFSv4 ACL settings, file operations such as create, delete, read, write, and permission changes can be recorded as ONTAP audit logs in EVTX or XML format.

• Delivery: ONTAP writes rotated audit log files to an audit volume inside the SVM
• Access path: Lambda reads those files through an FSx for ONTAP S3 Access Point
• Trigger: EventBridge Scheduler invokes Lambda periodically; Lambda uses checkpointing to process newly rotated files
• Compute: Lambda (scheduled, pay-per-invocation)
• Latency: Near-real-time rather than sub-second streaming. End-to-end latency depends on your ONTAP audit log rotation interval and the EventBridge Scheduler frequency.
• Use case: Compliance auditing, access pattern analysis, data governance

2. EMS (Event Management System) Webhooks

ONTAP’s built-in event system can push critical alerts via HTTP webhooks. This includes:

• Autonomous Ransomware Protection (ARP) alerts — ONTAP detects encryption patterns and fires an event
• Quota threshold violations
• Hardware failures

• Replication issues

• Delivery: ONTAP pushes HTTPS webhook to API Gateway

• Trigger: API Gateway invocation (event-driven)

• Compute: Lambda (behind API Gateway)

• Use case: Security alerting, operational monitoring

3. FPolicy (File Policy) Events

FPolicy intercepts file operations at the protocol level (CIFS/NFS) and forwards them in real-time via a proprietary TCP protocol. Unlike the other two sources, FPolicy requires a persistent TCP listener — which is why this path uses ECS Fargate rather than Lambda.

• Delivery: ONTAP connects to Fargate task via TCP:9898
• Trigger: Fargate receives FPolicy events → enqueues to SQS → Lambda processes
• Compute: ECS Fargate (TCP listener) + Lambda (vendor shipping)
• Use case: File activity monitoring, DLP, suspicious behavior detection

Note: The FPolicy path is the one exception to the “pure Lambda” model. ONTAP’s FPolicy protocol is a proprietary binary format over TCP — it cannot be received by API Gateway or Lambda directly. Fargate handles the protocol translation, then hands off to Lambda via SQS for the vendor-specific shipping. It’s still EC2-free, but not entirely serverless in the strictest sense.

The Architecture

Each event source feeds into the same delivery pattern:

┌─────────────────────────────────────────────────────────────────┐
│                    FSx for ONTAP                                │
├──────────────┬──────────────────────┬───────────────────────────┤
│ File Access  │   EMS Webhook        │   FPolicy (TCP:9898)      │
│ Audit Logs   │                      │                           │
└──────┬───────┴──────────┬───────────┴───────────┬───────────────┘
       │                  │                       │
       ▼                  ▼                       ▼
  FSx S3 AP +        API Gateway            ECS Fargate
  Scheduler               │                       │
       │                  ▼                       ▼
       ▼             Lambda (EMS)           SQS → Lambda
  Lambda (parser)         │                       │
       │                  │                       │
       └──────────────────┼───────────────────────┘
                          ▼
              Observability Vendor API
              (Datadog, Splunk, New Relic, ...)

Each integration packages the parser and vendor shipper together in a single Lambda, but the pattern is the same: normalize ONTAP events, then send them to the vendor API. Swap the integration Lambda, and you switch vendors. Vendor-specific Lambdas are optimized for quick adoption and native API behavior, while the OpenTelemetry integration provides a vendor-neutral path for organizations standardizing on OTLP.

The Gotcha That Cost Me a Day

Here’s something that isn’t immediately obvious from the documentation:

In my validation, a Lambda function placed in a VPC with only an S3 Gateway Endpoint could not read from the FSx for ONTAP S3 Access Point and timed out. Adding NAT Gateway egress resolved the issue.

This gotcha matters because this project intentionally reads audit logs through FSx for ONTAP S3 Access Points rather than mounting the audit volume over NFS/SMB from an EC2 instance.

Tested with:

• Lambda in private subnets (ap-northeast-1)
• FSx for ONTAP S3 Access Point attached to an FSx volume
• S3 Gateway VPC Endpoint only
• No NAT Gateway
• Failure mode: timeout (no response, not AccessDenied)

Your options:

This is based on my validation environment (ap-northeast-1). Always test the network path in your own account and Region, as AWS may update this behavior.

Target Vendors

The project targets 9 observability platforms. Datadog is fully verified end-to-end (the subject of Parts 2 and 3 of this series). The remaining vendors have initial implementations that I’ll be verifying and writing about in upcoming posts:

Status definitions:

• ✅ E2E verified — Deployed and validated with real FSx for ONTAP audit logs
• 🧪 Implementation ready — Code and CloudFormation available; E2E validation pending
• 🚧 Planned — Design exists; implementation pending

Each vendor integration is designed as a self-contained CloudFormation stack with its own Lambda, IAM roles, DLQ, and CloudWatch alarms. As I verify each one, I’ll publish a dedicated article with the results and any vendor-specific gotchas I encounter.

What’s in the Repo

The project is structured for easy adoption:

fsxn-observability-integrations/
├── integrations/
│   ├── datadog/           # ✅ Verified: Lambda + CFn + tests + docs
│   ├── splunk-serverless/ # 🧪 Implementation ready
│   ├── new-relic/         # 🧪 Implementation ready
│   ├── grafana/           # 🧪 Implementation ready
│   ├── elastic/           # 🧪 Implementation ready
│   ├── dynatrace/         # 🧪 Implementation ready
│   ├── sumo-logic/        # 🧪 Implementation ready
│   ├── honeycomb/         # 🧪 Implementation ready
│   └── otel-collector/    # 🧪 Implementation ready
├── shared/
│   ├── lambda-layers/     # Reusable log parser (EVTX/XML) + S3 AP reader
│   ├── templates/         # Prerequisites CFn (EventBridge Scheduler, IAM)
│   └── scripts/           # Deploy + test utilities
└── docs/                  # Bilingual (EN/JA) documentation

The shared infrastructure (EventBridge Scheduler, log parser layer, IAM roles) is vendor-agnostic and already proven through the Datadog verification. Each vendor directory follows the same structure, so once you understand one, you understand them all. Each stack is designed to include DLQ, CloudWatch alarms, and operational visibility out of the box; the Datadog stack also includes the verified CloudWatch operational dashboard used during E2E validation.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Related Posts

If you’ve been following my FSx for ONTAP S3 Access Points series, this project builds directly on those foundations:

• FSx for ONTAP S3 Access Points as a Serverless Automation Boundary — Where this journey started: using S3 APs as the bridge between ONTAP and serverless
• Production-Ready FPolicy Event Pipeline Across 17 UCs — Phase 11 — The FPolicy pipeline that feeds into this observability project
• Near-Real-Time Processing, ML Inference, and Observability — Phase 3 — Early architecture patterns that evolved into this multi-vendor approach

This observability integrations project is the natural next step: taking those serverless patterns and applying them specifically to audit log shipping across multiple vendors.

Design Considerations

Based on early feedback, here are key points for different audiences:

Design philosophy: The goal is not just to remove EC2. The goal is to move undifferentiated collector operations into managed services, make failures observable and replayable, and keep the integration layer small enough for customers to operate themselves.

Where this pattern matters: This pattern is especially useful for enterprise file workloads where auditability matters but EC2-based collectors add operational overhead —
departmental file shares, enterprise application interface directories such as SAP, Oracle, or SQL Server adjacent file shares, VDI/EUC home directories, engineering and design repositories, regulated file repositories, and ransomware investigation workflows.

Non-intrusive by design: This pipeline observes audit logs after ONTAP records them; it does not sit in the application data path. NFS/SMB access patterns are unchanged. No application code changes are required.

Telemetry ownership: This pattern treats ONTAP as the authoritative source of file activity telemetry, while AWS managed services provide the event processing and delivery layer.

Compliance note: This pattern helps centralize and analyze audit events, but retention, immutability, and regulatory controls should be designed according to your organization’s compliance requirements. This is an audit log delivery pattern, not a compliance certification. For audit evidence, consider separately how long raw EVTX/XML files should be retained on the audit volume or archived outside the observability pipeline.

Audit policy dependency: The quality and volume of events depend heavily on your ONTAP audit policy, SACLs, NFSv4 ACLs, and rotation interval. Enabling read auditing on high-traffic volumes can produce significant log volume — design your audit policy carefully.

Cost variables: The biggest cost factors are audit event volume, log rotation frequency, EventBridge Scheduler frequency, Lambda runtime, NAT Gateway usage (if Lambda is in VPC), and vendor ingest pricing. Compared to the EC2 pattern, you trade always-on instance cost for pay-per-invocation compute and vendor-ingest-driven cost.

Multi-account deployment: This pattern can be deployed per workload account or centralized into a logging/security account, depending on your organization’s landing zone design.

Reliability: The stack includes DLQ for failed events, CloudWatch alarms for error/throttle detection, and checkpointing to avoid reprocessing already completed audit log files. Delivery to external vendor APIs should be treated as at-least-once; DLQ messages can be replayed after resolving the root cause.

What’s Coming Next

This is Part 1 of a series. In the upcoming posts, I’ll deep-dive into:

• Part 2: Implementing the Datadog integration end-to-end — from CloudFormation to seeing logs in the Datadog Log Explorer
• Part 3: Event-driven ransomware detection using ONTAP’s Autonomous Ransomware Protection (ARP) + EMS webhooks + Datadog alerting

Beyond this Datadog series, I’ll be verifying and writing about each vendor integration as I go:

• Replacing the EC2-based Splunk pattern with Lambda + HEC
• OpenTelemetry as the vendor-neutral escape hatch
• Grafana Cloud + Loki for the open-source stack
• And more — each with its own E2E verification and lessons learned

The goal is to build a comprehensive, battle-tested pattern library where you can pick your vendor and deploy with confidence. Follow along as I work through each one.

Try It Yourself

The Datadog integration is fully verified and ready to deploy. You’ll need:

• An FSx for ONTAP file system with audit logging enabled
• An FSx for ONTAP S3 Access Point attached to the audit volume

git clone https://github.com/Yoshiki0705/fsxn-observability-integrations.git
cd fsxn-observability-integrations

# Deploy Datadog integration
# (FsxS3AccessPointArn = your FSx for ONTAP S3 Access Point ARN)
aws cloudformation deploy 
  --template-file integrations/datadog/template.yaml 
  --stack-name fsxn-datadog-integration 
  --parameter-overrides 
    FsxS3AccessPointArn=<your-fsx-s3-ap-arn> 
    DatadogApiKeySecretArn=<your-secret-arn> 
  --capabilities CAPABILITY_NAMED_IAM

This stack deploys the scheduled Lambda processor, IAM permissions for reading from the FSx for ONTAP S3 Access Point, checkpoint storage, DLQ, CloudWatch alarms, and the Datadog shipping logic. The processor keeps track of already-processed audit log files so each scheduled invocation only ships newly rotated logs.

After deployment, you should see:

• EventBridge Scheduler invoking the Lambda processor on your configured interval
• Checkpoint storage updated after processing rotated audit logs
• Parsed FSx for ONTAP audit events arriving in Datadog Logs (source:fsxn)
• CloudWatch alarms and DLQ ready for operational visibility

Full setup guide in the repo’s Prerequisites doc.

Have questions or want to see a specific vendor integration verified next? Drop a comment below — it’ll help me prioritize the series.

Next up: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way

Vue 3 launched in late 2020. Five years later, Vue 2 has been officially end-of-life since December 2023, yet migration is still dragging in many projects. Between the reactivity rewrite, the Composition API that throws people off at first, and Vite replacing webpack — there’s plenty of reason to stall. Here’s what actually changed, what it means for your code, and how to choose between the two APIs.

Vue 2 vs Vue 3: what changed in practice

The reactivity system

This is the deepest change. Vue 2 uses Object.defineProperty to observe mutations — with its well-known limitations: no detection of dynamically added properties, no detection on array indexes. Hence the Vue.set() and this.$set() calls that clutter Vue 2 code.

Vue 3 uses ES2015 Proxy. The entire object is intercepted, not property by property. No more $set, direct mutations are detected natively:

// Vue 2 — adding a reactive property after creation
this.$set(this.user, 'email', '[email protected]') // required

// Vue 3 — works directly
this.user.email = '[email protected]' // reactive automatically

Performance and bundle size

Vue 3 is tree-shakable. Unused features (Transition, KeepAlive, Teleport…) are not included in the final bundle if they aren’t imported. Vue 2 included the full runtime regardless.

In practice on an average project: runtime bundle reduced by roughly 40% with Vue 3 vs Vue 2. The reactive diff is also faster thanks to a rewritten virtual DOM patching strategy (static template analysis at compile time).

TypeScript

Vue 2 had a patched-together TypeScript support via vue-class-component and decorators. Functional, but tedious. Vue 3 was rewritten in TypeScript from scratch — types are native, inference works without any special configuration in components.

Fragments, Teleport, Suspense

Three structural additions that were missing in Vue 2:

• Fragments: a component can return multiple root elements — no more pointless wrapper <div>
• Teleport: render HTML into another DOM node (modals, tooltips) without working around CSS with fragile position: fixed hacks
• Suspense: handle async component loading states natively

What breaks during migration

The Vue 3 breaking changes that cause the most damage during migration:

• $listeners removed — merged into $attrs
• $children removed — use ref or provide/inject
• filters removed — use methods or computed properties
• v-model: the event is now update:modelValue instead of input
• Vuex → Pinia (Vuex 5 never shipped)
• Vue Router 4 with some API changes

On a large Vue 2 project with lots of filters, $listeners, and Vuex everywhere, migration takes time. The automated migration via @vue/compat helps but doesn’t solve everything.

Options API vs Composition API

This is the question that comes up most often when moving to Vue 3. The short answer: both are officially supported, permanently. It’s not a replacement but an alternative.

Options API — quick reminder

export default {
  name: 'UserProfile',
  props: {
    userId: String,
  },
  data() {
    return {
      user: null,
      loading: false,
    }
  },
  computed: {
    displayName() {
      return this.user?.name ?? 'Unknown'
    },
  },
  methods: {
    async fetchUser() {
      this.loading = true
      this.user = await api.getUser(this.userId)
      this.loading = false
    },
  },
  mounted() {
    this.fetchUser()
  },
  watch: {
    userId(newId) {
      this.fetchUser()
    },
  },
}

Composition API — same component

import { ref, computed, watch, onMounted } from 'vue'
import { useUserApi } from '@/composables/useUserApi'

export default {
  name: 'UserProfile',
  props: {
    userId: String,
  },
  setup(props) {
    const user = ref(null)
    const loading = ref(false)

    const displayName = computed(() => user.value?.name ?? 'Unknown')

    async function fetchUser() {
      loading.value = true
      user.value = await api.getUser(props.userId)
      loading.value = false
    }

    watch(() => props.userId, fetchUser)
    onMounted(fetchUser)

    return { user, loading, displayName }
  },
}

Or with <script setup> (the recommended syntax in Vue 3):

// <script setup>
import { ref, computed, watch, onMounted } from 'vue'

const props = defineProps({ userId: String })
const user = ref(null)
const loading = ref(false)

const displayName = computed(() => user.value?.name ?? 'Unknown')

async function fetchUser() {
  loading.value = true
  user.value = await api.getUser(props.userId)
  loading.value = false
}

watch(() => props.userId, fetchUser)
onMounted(fetchUser)

Everything declared inside <script setup> is automatically exposed to the template — no more return {}.

Strengths and weaknesses

Options API

Strengths

• Rigid and predictable structure — everyone knows where to find data, methods, computed properties
• Low learning curve — ideal for onboarding junior devs or devs coming from other frameworks
• Very readable on small components — everything is organized by option type
• Abundant documentation, examples everywhere on Stack Overflow and legacy projects

Weaknesses

• Business logic gets fragmented across data, methods, computed, watch — impossible to keep one “feature” coherent in a single block
• Reusing logic between components forces you through mixins — which create naming collisions and make dependencies opaque
• Tedious TypeScript: this inside methods doesn’t infer well, annotations everywhere
• On large components (200+ lines), navigation becomes painful: data is at the top, the method that modifies it is at the bottom, the watcher is somewhere else

Composition API

Strengths

• Business logic can be co-located: everything related to fetchUser (state, method, watcher) stays together
• Composables cleanly replace mixins: reusable logic is an explicitly imported function, no magic, no collisions
• Native TypeScript: ref<User | null>(null), no this, full inference
• More effective tree-shaking: only the Vue functions you use are imported

Weaknesses

• More verbose on small components — importing ref, computed, onMounted explicitly for 20 lines of logic
• The .value on refs is constant friction, a source of bugs (forgetting .value in JS but not in the template)
• Without discipline, <script setup> can turn into a grab-bag — structural freedom has a cost if the team doesn’t enforce conventions
• Less intuitive for someone coming from Vue 2 or React Class Components — the learning curve is real

Direct comparison on concrete cases

Logic reuse: mixins vs composables

This is where the Composition API wins most clearly. A Vue 2 mixin to handle pagination:

// Vue 2 — mixin
export const paginationMixin = {
  data() {
    return {
      page: 1,
      perPage: 10,
      total: 0,
    }
  },
  computed: {
    totalPages() {
      return Math.ceil(this.total / this.perPage)
    },
  },
  methods: {
    nextPage() { this.page++ },
    prevPage() { this.page-- },
  },
}

// Usage — where data.page comes from is not obvious
export default {
  mixins: [paginationMixin],
  // where does this.page come from? mystery for a new dev
}

// Vue 3 — composable
export function usePagination(initialPerPage = 10) {
  const page = ref(1)
  const perPage = ref(initialPerPage)
  const total = ref(0)

  const totalPages = computed(() => Math.ceil(total.value / perPage.value))

  function nextPage() { page.value++ }
  function prevPage() { page.value-- }

  return { page, perPage, total, totalPages, nextPage, prevPage }
}

// Usage — explicit
const { page, total, nextPage } = usePagination(20)
// we know exactly where page comes from

TypeScript: the concrete difference

// Options API with TypeScript — tedious
export default defineComponent({
  data(): { user: User | null; loading: boolean } { // manual annotation
    return { user: null, loading: false }
  },
  methods: {
    async fetchUser(id: string): Promise<void> {
      this.user = await getUser(id) // this isn't always inferred
    },
  },
})

// Composition API — native inference
const user = ref<User | null>(null)
const loading = ref(false) // inferred as ref<boolean>

async function fetchUser(id: string) {
  user.value = await getUser(id) // correctly typed without annotation
}

Conclusion: when to choose which

Options API if:

• You’re migrating a Vue 2 project and want to minimize change
• The team is junior or comes from other MVC frameworks
• Components are small and logic isn’t reused
• You want a framework-enforced structure without the discipline overhead

Composition API if:

• You’re starting a new Vue 3 project — it’s the ecosystem’s default choice
• You’re using TypeScript seriously
• You have complex logic to reuse across components
• Components are growing — logic co-location becomes a real advantage

What I wish I’d known when starting with Vue 3: both APIs coexist in the same project. A simple form component can stay in Options API while a complex component with 5 composables switches to Composition API. No need to choose once and for all.

On my recent projects, I use the Composition API by default with <script setup> — TypeScript demands it — and composables for everything touching API calls, complex local state management, and DOM interactions (resize observer, intersection observer, etc.). Options API stays for simple presentational components where the rigid structure is an advantage rather than a constraint.

PKM, RAG, wikis, and AI memory systems are often discussed as if they solve the same problem.
They do not.
They all deal with knowledge, but they operate at different layers:

• PKM helps humans think.
• Wikis help groups preserve shared knowledge.
• RAG helps machines retrieve external knowledge.
• Memory systems help AI agents persist context over time.

Confusing these systems leads to bad architecture.

You get wikis full of personal scratch notes, RAG systems without a source of truth, memory layers pretending to be databases, and PKM tools overloaded with automation they were never designed to handle.

A better model is to see them as different parts of a knowledge systems spectrum.

This article compares PKM, RAG, wikis, and AI memory systems by structure, retrieval, ownership, evolution, and real-world use cases.

The short version

The most important distinction is this:

PKM and wikis structure knowledge. RAG retrieves knowledge. Memory systems evolve agent context.

That is the core mental model.

Why these systems are confused

They overlap in visible behavior.

All of them can:

• store notes
• retrieve information
• answer questions
• organize references
• connect ideas

But they differ in intent.

A PKM system is not just a private wiki.
A wiki is not just a RAG database.
A RAG pipeline is not an AI memory.
An AI memory system is not a replacement for structured documentation.

The confusion comes from treating “knowledge” as one thing.

In practice, knowledge has multiple layers:

1. Capture
2. Structure
3. Retrieval
4. Interpretation
5. Reuse
6. Evolution

Different systems optimize different stages.

The four paradigms

1. PKM

PKM stands for personal knowledge management.

It is the practice of capturing, organizing, connecting, and using knowledge for personal work.

Typical PKM systems include:

• Obsidian
• Logseq
• Notion
• plain Markdown folders
• Zettelkasten systems
• second brain systems

PKM is human driven.

The goal is not just storage. The goal is better thinking.

What PKM is good at

PKM works well for:

• learning a new domain
• developing original ideas
• connecting notes over time
• writing articles or books
• tracking personal research
• building a second brain

A good PKM system is messy in a useful way. It supports unfinished thoughts, partial ideas, private context, and evolving concepts.

This is why PKM is not the same as documentation.

Documentation wants clarity.
PKM tolerates ambiguity.

PKM failure modes

PKM often fails when it becomes:

• a dumping ground
• a folder taxonomy project
• a productivity aesthetic
• a tool optimization hobby
• a private archive nobody uses

The main risk is collection without synthesis.

If you only save information, you do not have a knowledge system. You have a personal landfill.

Opinionated take

PKM should optimize for reuse, not capture.

Capturing everything feels productive, but it creates debt. The real value appears when notes become connected, rewritten, compressed, and used in output.

2. Wiki

A wiki is a structured knowledge base designed for shared reference.

Typical wiki systems include:

• DokuWiki
• MediaWiki
• Confluence
• BookStack
• Git based documentation sites
• internal company knowledge bases

A wiki is usually more formal than PKM.

It should answer:

What do we know, and where is the current version?

What wikis are good at

Wikis work well for:

• team documentation
• operational runbooks
• product knowledge
• policy documents
• technical reference
• onboarding material
• stable domain knowledge

A wiki is a social contract.

It says:

This page is the place where this knowledge lives.

That makes ownership and maintenance critical.

Wiki failure modes

Wikis often fail because they become stale.

Common problems:

• no page owners
• outdated screenshots
• duplicate pages
• unclear canonical versions
• too much hierarchy
• no maintenance rhythm

A wiki with old information is worse than no wiki, because it creates false confidence.

Opinionated take

A wiki should be boring.

That is a compliment.

A good wiki is not where ideas are born. It is where stable knowledge is preserved after it becomes useful to others.

3. RAG

RAG stands for retrieval augmented generation.

It is an AI architecture where a system retrieves relevant external information before asking a language model to generate an answer.

A basic RAG pipeline usually has:

1. Documents
2. Chunking
3. Embeddings or search index
4. Retrieval
5. Optional reranking
6. Prompt assembly
7. LLM generation

RAG is machine driven.

The goal is not to create knowledge. The goal is to give a model relevant context at query time.

What RAG is good at

RAG works well for:

• question answering over documents
• internal search assistants
• support bots
• technical documentation assistants
• compliance lookup
• research over large corpora
• connecting LLMs to updated information

RAG is especially useful when the model cannot or should not memorize the information.

RAG failure modes

RAG often fails when teams treat it as magic search.

Common problems:

• bad chunking
• weak retrieval
• noisy context
• missing metadata
• no source of truth
• stale documents
• weak evaluation
• no human feedback loop

RAG does not fix bad knowledge management.

If the underlying content is fragmented, outdated, or contradictory, the RAG system will surface that mess with confidence.

Opinionated take

RAG is not a knowledge strategy.

RAG is an access strategy.

It helps machines access knowledge, but it does not decide what knowledge is valid, maintained, canonical, or useful.

4. AI memory systems

AI memory systems give agents persistent context beyond a single prompt or conversation.

They may store:

• user preferences
• past decisions
• long-term facts
• task history
• summaries
• reflections
• extracted entities
• episodic memories
• semantic memories

Examples and related ideas include:

• MemGPT style memory tiers
• long-term agent memory
• episodic memory
• semantic memory
• vector memory
• profile memory
• tool state memory
• reflective agents

AI memory is agent driven.

The goal is continuity.

What AI memory is good at

AI memory systems work well for:

• personal assistants
• long-running coding agents
• research agents
• customer support agents
• tutoring systems
• workflow automation
• persistent companions
• multi-session task execution

Memory matters when the system must behave as if it remembers.

AI memory failure modes

Memory systems are dangerous when unmanaged.

Common problems:

• remembering wrong facts
• storing too much
• privacy risk
• stale preferences
• poor memory ranking
• memory poisoning
• no forgetting mechanism
• confusing memory with truth

A memory system needs governance.

It should answer:

• What should be remembered?
• Who approved it?
• How long should it live?
• When should it be forgotten?
• How is it corrected?

Opinionated take

AI memory is not just long context.

Long context lets a model see more at once.
Memory decides what survives across time.

Those are different problems.

Core differences table

Structure vs retrieval vs evolution

The simplest way to understand these systems is to compare what they optimize. The architectural implications of that distinction are explored in depth in Retrieval vs Representation in Knowledge Systems.

PKM optimizes personal evolution

PKM is about how your understanding changes.

You collect material, rewrite it, connect it, and turn it into something useful.

The output is often:

• a better mental model
• a written article
• a decision
• a research direction
• a reusable insight

PKM is not primarily about fast lookup. It is about long-term sensemaking.

Wikis optimize shared structure

Wikis are about stable knowledge.

They ask:

• What is the current answer?
• Who owns it?
• Where should people go?
• What should be updated?

A wiki works when people trust it.

RAG optimizes machine retrieval

RAG is about retrieving the right context at the right time.

It asks:

• What documents are relevant?
• Which chunks should be used?
• How much context fits?
• What should the model cite?

RAG works when retrieval quality is high and the source corpus is trustworthy.

AI memory optimizes continuity

Memory systems are about persistence across sessions.

They ask:

• What should the agent remember?
• What should be forgotten?
• Which memory matters now?
• How should memory change behavior?

Memory works when it improves future behavior without polluting the agent with stale or incorrect context.

When to use PKM

Use PKM when the knowledge is personal, unfinished, or exploratory.

Good scenarios:

• learning distributed systems
• planning articles
• researching LLM architecture
• collecting book notes
• building a second brain
• tracking personal experiments

Use PKM when you are still thinking.

Example

You are learning about RAG evaluation.

You collect:

• articles
• benchmark notes
• diagrams
• implementation ideas
• failures from your own experiments

This belongs in PKM first.

Later, once the knowledge stabilizes, you may publish an article or turn it into documentation.

When to use a wiki

Use a wiki when knowledge must be shared and maintained.

Good scenarios:

• team onboarding
• API documentation
• operational runbooks
• architecture decision records
• product knowledge
• deployment instructions
• support procedures

Use a wiki when others need a reliable answer.

Example

Your team has one correct way to deploy a Hugo site to S3 and CloudFront.

That does not belong only in someone’s private notes.

It belongs in a wiki or documentation system with clear ownership.

When to use RAG

Use RAG when an AI system needs access to external knowledge at query time.

Good scenarios:

• chatbot over documentation
• search assistant over internal docs
• support assistant over help articles
• legal or compliance assistant
• research over large document sets
• developer assistant over code docs

Use RAG when the problem is:

The model needs information that lives outside its weights.

Example

You have hundreds of technical articles and want an assistant to answer questions using them.

RAG is a good fit.

But only if the documents are clean enough to retrieve from.

When to use AI memory

Use AI memory when an agent needs continuity.

Good scenarios:

• coding agents that remember project conventions
• personal assistants that remember preferences
• research agents that continue long investigations
• tutoring agents that remember student progress
• support agents that remember prior interactions
• autonomous agents that track goals

Use memory when the system must improve across time.

Example

A coding agent should remember:

• the project uses Go
• tests run with a specific command
• the user prefers minimal dependencies
• database migrations follow a convention

That is not just retrieval. It is persistent operating context.

How these systems combine

The most useful systems are hybrids.

A mature knowledge architecture might look like this:

1. PKM for personal exploration
2. Wiki for stable shared knowledge
3. RAG for machine access
4. AI memory for long-running agent continuity

Each layer has a job.

Pattern 1. PKM to wiki

This is the human knowledge pipeline.

Flow:

1. Capture notes privately
2. Connect ideas
3. Distill insights
4. Publish stable knowledge
5. Maintain as shared reference

This is how personal research becomes organizational knowledge.

Example

You research self-hosted knowledge tools in Obsidian.

After testing DokuWiki, Nextcloud, and static Markdown systems, you write a stable guide in your site or team wiki.

PKM created the insight.
The wiki preserves the result.

Pattern 2. Wiki to RAG

This is the machine access pipeline.

Flow:

1. Maintain canonical wiki pages
2. Index them
3. Retrieve relevant sections
4. Generate grounded answers
5. Link back to sources

This is one of the cleanest RAG patterns.

The wiki remains the source of truth.
RAG becomes the access layer.

Example

A support bot answers questions using a product wiki.

The bot should not replace the wiki. It should cite and route users back to the canonical pages.

Pattern 3. RAG plus memory

This is the agent continuity pipeline.

Flow:

1. RAG retrieves external facts
2. Memory stores user or task context
3. The agent combines both
4. Future behavior improves

RAG answers:

What does the knowledge base say?

Memory answers:

What matters about this user, project, or task?

Example

A coding agent uses RAG to retrieve framework docs.

It uses memory to remember that your project avoids ORMs, prefers sqlc, and uses structured logging.

Those are different knowledge types.

Pattern 4. PKM plus AI assistant

This is the hybrid thinking pipeline.

Flow:

1. Human captures notes
2. AI summarizes and suggests links
3. Human edits and validates
4. Knowledge becomes more structured
5. Some pages graduate to wiki or publication

The AI augments the PKM system, but it should not own the truth.

Example

An AI assistant can suggest connections between notes about RAG, memory systems, and LLM Wiki.

But the human decides which connections are meaningful.

Common architecture mistakes

Mistake 1. Treating RAG as a wiki

RAG is not a knowledge base.

It does not automatically create a canonical structure. It retrieves from whatever exists.

If the source documents are bad, RAG becomes a confident interface to bad knowledge.

Mistake 2. Treating memory as a database

AI memory is selective context, not general storage.

A database stores records.
Memory changes behavior.

If you need exact facts, use a database or knowledge base.
If you need continuity, use memory.

Mistake 3. Treating PKM as documentation

PKM can be messy.

Documentation should not be.

Private notes can contain half-formed ideas. Shared documentation should contain stable, maintained knowledge.

Mistake 4. Treating a wiki as a thinking tool

A wiki can support thinking, but it is not ideal for early exploration.

If every early thought must become a polished page, people stop writing.

Use PKM for rough thinking. Use wikis for durable knowledge.

Mistake 5. Treating long context as memory

Long context is not memory.

It only helps while the context is present.

Memory persists, selects, updates, and sometimes forgets.

Decision guide

Use this simple decision model.

If the knowledge is private and evolving

Use PKM.

If the knowledge is shared and stable

Use a wiki.

If an AI needs to answer from external documents

Use RAG.

If an agent needs continuity over time

Use memory.

If you need all four

Build a layered system.

Do not force one tool to do every job.

The knowledge systems spectrum

These systems form a spectrum from human thinking to AI continuity.

The direction matters.

Knowledge often starts as personal thought, becomes shared structure, is indexed for machine retrieval, and then becomes part of persistent agent behavior.

That is the modern knowledge stack.

Where LLM Wiki fits

LLM Wiki style systems sit between wiki and AI architecture.

They are not classic RAG.

Instead of retrieving chunks only at query time, they attempt to pre-structure knowledge into pages, summaries, entities, and links.

That makes them closer to compiled knowledge systems.

A useful placement:

This is why LLM Wiki belongs near knowledge systems architecture, not inside ordinary RAG.

Practical examples

Example 1. Personal technical blog

A technical blogger might use:

• PKM for research notes
• Hugo site as published knowledge
• internal linking as wiki-like structure
• RAG later for site search
• AI memory for writing assistant preferences

This is a strong architecture.

It keeps human judgment at the center while still allowing AI support.

Example 2. Engineering team

An engineering team might use:

• PKM for individual learning
• wiki for standards and runbooks
• RAG assistant for internal docs
• memory for coding agents working inside repositories

The wiki should remain canonical.

The RAG assistant should not invent process.
The memory layer should remember project preferences, not replace architecture decisions.

Example 3. AI research workflow

A researcher might use:

• PKM for paper notes
• wiki for stable summaries
• RAG for literature search
• memory for long-running research agents

This works because each layer handles a different time scale.

Security and governance

Knowledge systems become risky when they store sensitive or stale information.

PKM governance

Questions:

• What should stay private?
• What should be published?
• What should be deleted?

Wiki governance

Questions:

• Who owns each page?
• When was it last reviewed?
• What is canonical?

RAG governance

Questions:

• Which sources are indexed?
• Are answers cited?
• How is retrieval evaluated?
• What content is excluded?

Memory governance

Questions:

• What is remembered?
• Can users inspect memory?
• Can users delete memory?
• How are wrong memories corrected?

Memory needs the strictest governance because it can silently influence future behavior.

SEO and content strategy note

If you run a technical site, this distinction is not only architectural. It is also editorial.

You can map content like this:

• PKM pages explain human knowledge practices.
• Wiki pages explain structured knowledge systems.
• RAG pages explain retrieval engineering.
• Memory pages explain persistent AI behavior.
• Architecture pages compare and connect the paradigms.

This gives your site a clean authority mesh instead of a pile of loosely related AI articles.

Final conclusion

PKM, RAG, wikis, and AI memory systems are not competitors.

They are different answers to different questions.

PKM asks:

How do I think better over time?

A wiki asks:

What do we know, and where is the trusted version?

RAG asks:

What external context should the model use right now?

AI memory asks:

What should this agent remember for the future?

Once you separate those questions, the architecture becomes obvious.

Use PKM for thinking.
Use wikis for shared truth.
Use RAG for retrieval.
Use memory for continuity.

The future is not one knowledge system that replaces all others.

The future is layered knowledge architecture. For tools, methods, and self-hosted platforms across the full knowledge management spectrum, the cluster pillar maps the territory.

Sources and further reading

• https://cloud.google.com/use-cases/retrieval-augmented-generation
• https://aws.amazon.com/what-is/retrieval-augmented-generation/
• https://www.ibm.com/think/topics/retrieval-augmented-generation
• https://www.ibm.com/think/topics/knowledge-management
• https://arxiv.org/abs/2310.08560
• https://research.memgpt.ai/
• https://zettelkasten.de/posts/building-a-second-brain-and-zettelkasten/