Notifications start small. “Send the user an email when their order ships.” A function. A library. Done.

A year later, you have email, SMS, push, in-app, Slack, and Microsoft Teams. You have user preferences per channel. You have quiet hours, batching, throttling, and a “do not disturb” mode. You have unsubscribe links and bounce handling. You have analytics on open rates and template-level metrics. You have multi-language templates and timezone-aware scheduling.

What started as a function is now a system. If you keep it as a sprawling collection of sendEmail and sendSlack helpers across your codebase, that system will eat your engineering team alive.

This is the shape of the notification microservice I built (and have rebuilt twice). The pattern isn’t novel — it’s domain-driven design applied to notifications — but the specifics matter.

The core insight

A notification has three distinct concerns:

1. What happened in the business — “order placed,” “user mentioned,” “invoice overdue.” This is a domain event.
2. What kind of message to send — “transactional email,” “high-urgency push,” “Slack mention.” This is a delivery policy.
3. How to actually send it — “render this template, then call the email provider’s API.” This is a channel adapter.

Most codebases collapse all three into one function:

async function sendOrderShippedEmail(orderId: string, userId: string) {
  const user = await getUser(userId)
  const order = await getOrder(orderId)
  const html = renderTemplate('order-shipped', { user, order })
  await sendgrid.send({ to: user.email, subject: 'Your order shipped', html })
}

This function knows about the domain event, the message type, and the channel. Three concerns. One function. Each one will change for different reasons, and changes will ripple across all the call sites.

The DDD-shaped version separates them.

Domain events

The business code emits an event. It doesn’t know or care how the notification gets delivered.

// In your order service
await eventBus.publish({
  type: 'order.shipped',
  occurredAt: new Date().toISOString(),
  data: {
    orderId: order.id,
    userId: order.userId,
    trackingNumber: order.trackingNumber,
  },
})

This is a fire-and-record operation. The order service is done. It moves on. The event lands in your event bus (BullMQ, Kafka, NATS, whatever).

The notification service consumes events. Its job is to translate “order.shipped” into “a transactional email to the user with this template.”

Notification preferences

The user’s preferences live in a separate domain. They might be stored in the notification service’s database or in a profile service — doesn’t matter, as long as they’re queryable:

type UserNotificationPreferences = {
  userId: string
  channels: {
    email: { enabled: boolean; address: string }
    sms: { enabled: boolean; number?: string }
    push: { enabled: boolean; deviceTokens: string[] }
    slack: { enabled: boolean; userId?: string }
  }
  perEventType: Record<string, {
    channels: ('email' | 'sms' | 'push' | 'slack')[]
    enabled: boolean
  }>
  quietHours: { start: string; end: string; tz: string } | null
}

When the notification service receives order.shipped, it looks up the user’s preferences. The user has email enabled, SMS enabled, push enabled — but for this event type (order.shipped), they’ve only chosen email. So the service sends one email.

This decoupling is crucial. The business code emits one event. The notification service decides what to do with it based on user preferences. The user can change their preferences without anyone touching the business code.

The dispatcher

The middle layer is a dispatcher that:

1. Consumes an event.
2. Looks up the user’s preferences.
3. Decides which channels to deliver on.
4. For each channel, builds a delivery job and queues it.

type Channel = 'email' | 'sms' | 'push' | 'slack'

class NotificationDispatcher {
  async handle(event: DomainEvent) {
    const userId = (event.data as any).userId
    if (!userId) return

    const prefs = await this.prefsRepo.findByUserId(userId)
    if (!prefs) return

    const eventPrefs = prefs.perEventType[event.type]
    if (!eventPrefs?.enabled) return

    const now = new Date()
    const channels = this.filterByQuietHours(eventPrefs.channels, prefs, now)

    for (const channel of channels) {
      await this.deliveryQueue.add('deliver', {
        eventType: event.type,
        userId,
        channel,
        eventData: event.data,
        scheduledAt: now.toISOString(),
      })
    }
  }

  private filterByQuietHours(
    requested: Channel[],
    prefs: UserNotificationPreferences,
    now: Date
  ): Channel[] {
    if (!prefs.quietHours) return requested
    const isQuiet = isWithinQuietHours(now, prefs.quietHours)
    if (!isQuiet) return requested
    // During quiet hours, only allow non-disruptive channels (e.g., email/in-app)
    return requested.filter(c => c === 'email' || c === 'in-app')
  }
}

The dispatcher is pure routing logic. It doesn’t render templates. It doesn’t call any provider. It just figures out which channels to deliver on and queues delivery jobs.

Channel adapters

Each channel is a separate worker that consumes jobs from the delivery queue and dispatches to its specific channel.

class EmailDeliveryWorker {
  async handle(job: DeliveryJob) {
    if (job.channel !== 'email') return

    const user = await this.userRepo.findById(job.userId)
    const prefs = await this.prefsRepo.findByUserId(job.userId)
    if (!user || !prefs?.channels.email.enabled) return

    const template = await this.templateRepo.find(job.eventType, 'email')
    const rendered = await this.renderer.render(template, {
      user,
      ...job.eventData,
    })

    await this.emailProvider.send({
      to: prefs.channels.email.address,
      from: rendered.from,
      subject: rendered.subject,
      html: rendered.html,
      text: rendered.text,
      headers: {
        'X-Event-Type': job.eventType,
        'X-User-Id': job.userId,
        'List-Unsubscribe': `<${this.unsubscribeUrl(job.userId, job.eventType)}>`,
      },
    })

    await this.deliveryLog.record({
      userId: job.userId,
      channel: 'email',
      eventType: job.eventType,
      sentAt: new Date().toISOString(),
      provider: this.emailProvider.name,
    })
  }
}

The email worker knows about:

• The user (to get their address).
• The template store (to find the right template for this event type and channel).
• The renderer (to fill in the template).
• The email provider (to actually send).

It doesn’t know about Slack, SMS, push, or any business logic. If I want to add a new channel, I add a new worker. The dispatcher already knows how to route to it (the channel is just a string in the job).

Templates as first-class entities

A template store is its own small domain. Each template has:

• An event type it’s for (order.shipped).
• A channel it’s for (email, sms).
• A language (en, de, fr).
• A version (so changes are auditable).
• The actual template content (HTML, plain text, Markdown).

type Template = {
  id: string
  eventType: string
  channel: Channel
  language: string
  version: number
  subjectTemplate?: string       // for email
  bodyTemplate: string
  format: 'html' | 'markdown' | 'plain'
  createdAt: string
  active: boolean
}

The renderer takes a template and a context object and produces a rendered message. Use a templating library that has a sandboxed mode (Handlebars’s strict mode, for example) — you do not want template authors writing arbitrary JS that gets executed during rendering.

class Renderer {
  async render(template: Template, context: Record<string, any>): Promise<Rendered> {
    const compiled = Handlebars.compile(template.bodyTemplate, { strict: true })
    const body = compiled(context)
    const subject = template.subjectTemplate
      ? Handlebars.compile(template.subjectTemplate, { strict: true })(context)
      : undefined
    return { body, subject, format: template.format }
  }
}

Templates being stored in a database (not in code) means non-engineers can edit them. We had marketing folks editing email copy via an admin panel without ever touching a deploy.

The unsubscribe surface

Every notification should be unsubscribable. The dispatcher checks enabled flags before queuing, but the user needs a way to flip those flags. Two patterns:

1. A preferences page in your app. Standard. Each event type has a checkbox per channel.
2. A one-click unsubscribe link in every notification. Required by law in many jurisdictions for email marketing, and good UX everywhere.

The unsubscribe link encodes the user ID, the event type, and a signed token. Clicking it flips the preference:

function unsubscribeUrl(userId: string, eventType: string): string {
  const token = sign({ userId, eventType, action: 'unsubscribe' })
  return `https://app.example.com/api/notify/unsubscribe?token=${token}`
}

The endpoint verifies the token, updates the preference, and shows a “you’re unsubscribed” page. The same endpoint can power the List-Unsubscribe header for email clients that support one-click unsubscribe.

Observability

Each delivery generates two records:

1. Delivery log. “We attempted to send X to user Y on channel Z at time T using provider P.”
2. Provider callback. “The provider says message M was delivered (or bounced, or opened, or clicked).”

Both feed into the same table, keyed by a message ID. The observability story collapses into “show me everything that happened for user Y this week,” which is what support teams ask for.

Throttling and batching

Two problems show up at scale:

1. A user gets 50 notifications in 10 minutes because something noisy happened. You need to batch them into a digest.
2. A celebrity user’s actions trigger 10,000 notifications to followers in a burst. You need to throttle.

The dispatcher is where this logic lives. Before queuing a delivery, check a sliding-window counter (Redis-backed):

async handle(event: DomainEvent) {
  // ...existing routing logic...

  for (const channel of channels) {
    const window = `notify:${userId}:${channel}:${eventType}`
    const count = await this.redis.incr(window)
    if (count === 1) await this.redis.expire(window, 60)  // 1-minute window

    if (count > MAX_PER_MINUTE) {
      // Throttled. Queue for batching instead.
      await this.batchQueue.add('batch', { userId, channel, eventType, eventData: event.data })
    } else {
      await this.deliveryQueue.add('deliver', { /* ... */ })
    }
  }
}

The batch worker runs periodically (every 5 minutes or whatever the digest schedule is), collects everything in the batch queue for a user, renders a “digest” template, and sends one combined notification.

What I’d warn the next team about

Don’t put rendering in the dispatcher. Keep the dispatcher routing-only. The dispatcher decides which channels to use; the workers decide how to render and send. Mixing them couples your routing logic to your template engine in ways that hurt later.

Use job retries with caution. If the email provider returns a 503, retry. If it returns a 400 with “invalid recipient,” do not retry — that’s a permanent failure. Different error codes have different retry semantics; encode this in the worker.

Track template performance. Open rate per template per language is a real signal. Templates that score below 10% open rate are usually broken (bad subject line, bad timing, irrelevant content). Surface this in your admin UI.

Make the dead-letter queue visible. Failed deliveries should go somewhere humans can see them. We had three months of bounces piling up before anyone noticed; turned out a customer had typo’d their email and was getting nothing. A weekly dashboard of “delivery failures by user” caught it.

The takeaway

A notification microservice is one of the highest-leverage extractions you can do. The business code becomes simple (emit events). User preferences become a first-class concept. Templates become editable without deploys. New channels become new workers, not new branches in shared functions.

The pattern is more code than sendgrid.send(...). It’s also the difference between “we have a notification feature” and “we have a notification platform.” If you’re shipping more than two notification types and you’re tired of touching the same five functions every time product adds a new channel, this extraction pays for itself within a quarter.

This is a submission for the Google I/O Writing Challenge

Google I/O 2026 brought incredible technical announcements, robust models, and massive context windows. But as someone who lives and breathes technology, I’ve come to realize that the greatest Artificial Intelligence revolution often isn’t in the number of parameters of a cloud-hosted model, but rather in how everyday people interact with it on the ground.

To test the true impact of this year’s updates, I decided to observe the ecosystem right inside my own home. My wife and my mother-in-law don’t work in tech and have always viewed these generative AI tools as something distant, complex, or purely “a programmer thing.”

Everything changed with the arrival of the new Gemini mobile interface, driven by the new design language and faster models presented at the event.

⚡ Fluidity and Speed: My Wife’s Experience

Previously, when someone tried to interact with an AI assistant on a phone, the response usually came in massive blocks of plain text, or with that “typing effect” that gave the distinct feeling of waiting for a machine to process data.

When my wife opened the new Gemini app and started talking to the AI, her reaction was immediate. The response came promptly and incredibly faster than any previous version.

Google’s big win here was refining the interface: the text now reconstructs itself in real time, prioritizing visual elements and scannable highlights. She even commented on how much more fluid and natural the interaction felt. The barrier to entry that used to feel like a “rigid oracle” vanished, making way for a real, dynamic, and extremely agile conversation.

📸 Going Beyond Text: The Camera Test with My Mother-in-Law

Another fantastic breakthrough from Google I/O was the evolution of real-time multimodality on mobile devices. For non-tech folks, the concept of “multimodality” sounds like Greek, but in daily practice, it translates to pure magic.

My mother-in-law decided to put this intelligence to the test in a very spontaneous way: she simply made a funny face at the camera while interacting with the app.

The AI’s response was instantaneous. Gemini not only captured the video feed flawlessly, but it identified her exact facial expression and joked about the situation. My mother-in-law was completely surprised. Seeing a technology understand the context of such a subtle human behavior—and doing it instantly—transformed her perception that AI is just a glorified text search engine.

🎯 The Takeaway: The Best Technology Is the One That Becomes Invisible

As developers, we love to discuss architecture, latency, and API updates. However, Google I/O 2026 reminded me of a valuable lesson about Product and User Experience (UX): it doesn’t matter if you have the smartest model in the world if the interface scares away the end user.

By focusing on an interface that responds fluidly and understands natural inputs like voice and images without friction, Google has finally broken out of the tech enthusiast bubble. Seeing Gemini transform into an accessible and fun assistant for my family is living proof that the future of AI is, above all, human.

Running a Full Multi-Stage Intrusion Simulation. Every Detection Fired.

I’ve been building SHENRON for a while — a polymorphic adversarial simulation framework that generates inert adversarial-shaped telemetry for detector and governance testing.

Last week I wired up real stealth scoring. Today I ran the apt_kill_chain scenario end to end and validated it against the full detection expectation suite. Here’s what happened.

What SHENRON Is

SHENRON doesn’t execute attacks. It simulates the shape of attacks — the behavioral signatures, telemetry patterns, and detection opportunities that real adversarial techniques produce, without the capability.

A layer is a self-contained behavioral simulation module with its own telemetry schema, ATT&CK mappings, and detection validation expectations. Each layer emits synthetic JSONL artifacts representing what a technique looks like to a detection system — nothing executable, nothing functional.

The safety contract enforced per-artifact:

simulation_only: true
executable: false
no_payload_present: true
network_calls_made: false
processes_spawned: false

The goal: give detection engineers and SOC teams a way to test whether their systems would see an attack — without running one.

The Scenario

Nine stages. Full ATT&CK coverage. Realistic inter-stage timing offsets.

[STAGE 1/9] initial_c2             — beacon_emitter_cloak      +0s
[STAGE 2/9] recon                  — lateral_webcrawler         +120s
[STAGE 3/9] persistence_plant      — dormant_sleeper_seed       +300s
[STAGE 4/9] memory_hijack          — memory_hijack_inheritor    +360s
[STAGE 5/9] cover_tracks           — anti_forensics_molt        +420s
[STAGE 6/9] masquerade             — mirror_loop_deflector      +450s
[STAGE 7/9] persistence_reinforce  — shadow_system_rebuilder    +480s
[STAGE 8/9] file_plant             — poltergeist_file_infector  +510s
[STAGE 9/9] exfil_c2               — beacon_emitter_cloak       +600s

ATT&CK coverage by stage:

Stage by Stage

Stage 1 — C2 Beacon Establishment

Three synthetic beacon events with realistic jitter:

method        : https_encoded        interval: 54.7s
dns_query_sim : mjcjbowe.metrics.example.io
method        : http_get             interval: 71.4s
user_agent    : curl/7.88.1

Stealth: 80. C2 over HTTPS with jittered intervals blends well. DNS subdomain entropy is the tell.

Detection signals: periodic outbound connection · DNS subdomain query · encoded URI parameter

Stage 2 — Lateral Reconnaissance

8 hosts discovered, 2 targeted for deeper enumeration:

172.16.1.112  BACKUPHOST   → port 8080 Tomcat → /login/ /. git/ /backup/
172.16.1.113  PRINTSERVER  → port 80 Apache  → /wp-admin/ /phpmyadmin/

Stealth: 78. Sequential host sweeps are noisy. The .git/ and /backup/ probes are high-signal for anomaly-based detection.

Detection signals: subnet sweep · SMB port probe · share enumeration · sequential host requests

Stage 3 — Persistence Plant

Trigger-armed persistence that waits for a condition before activating:

mechanism     : systemd_unit
path          : /etc/systemd/system/update-check.service
trigger_type  : port_open (localhost:4444)
check_interval: 45s
→ TRIGGER EVAL #4: condition met — activation simulated

Stealth: 65. Systemd unit creation is well-logged. Persistence layers consistently score lowest — they’re hard to hide from any reasonably instrumented environment.

Detection signals: scheduled task creation · registry run key write · cron modification

Stage 4 — Process Injection Simulation

/proc/6109/mem probed → stack region 0x7fff0000
injection_attempt: shellcode_stub_sim → success
token_manipulation: NetworkService impersonation
MONITOR CYCLE #3: runtime variance detected → anomaly: True

Stealth: 68. /proc/mem access from a non-parent process is a hard detection signal. The anomaly in cycle 3 is exactly what EDR behavioral engines catch.

Detection signals: proc mem access · token impersonation · RWX region write

Stage 5 — Anti-Forensics

~/.bash_history              → truncate_to_zero
~/.local/share/recently-used.xbel → overwrite_with_decoys
  decoys: ['cat /etc/hostname', 'cd /tmp', 'echo hello world']
mtime_rollback_24h on 3 targets

Stealth: 77. Log truncation is detectable via inotify or auditd. The decoy injection looks benign on cursory inspection — but timing correlation breaks it.

Detection signals: log file cleared · history truncated · timestamp rollback

Stage 6 — Process Masquerade

target_sim    : systemd-resolve (pid 5496)
cmdline_sim   : /usr/bin/systemd-resolve --daemon
DEFLECTION LOOP #3: anomaly logged

Stealth: 84 — highest in the chain. Name spoofing is hard to detect at the log level. The anomaly in loop 3 is what a behavioral engine catches: the process walks like systemd-resolve but doesn’t talk like it.

Detection signals: process name spoof · fake cmdline · PID masquerade

Stage 7 — Persistence Reinforcement

Self-healing persistence detects tampered files and restores from shadow copies:

/etc/rc.local                          hash mismatch → restored
/etc/profile.d/env.sh                  hash mismatch → restored
/etc/systemd/system/update.service     absent        → shadow copy deployed
ts_adjusted: True  (on all restores)

Stealth: 65. File integrity monitoring catches the restore. Timestamp adjustment is a second detectable signal in the same window.

Detection signals: file hash mismatch · unauthorized file copy · timestamp modification

Stage 8 — Script Modification

Recursive home traversal, candidate selection, EOF append:

scan: *.py *.sh across ~/
→ ~/research_hub/utils/scan.py selected
marker appended: # SYS_UPDATE_HOOK_SIM
hash_delta: 305ff87c... → a5fe9306...

Stealth: 68. Hash change on a user script is detectable by any FIM tool. The traversal is the noisy precursor.

Detection signals: recursive home traversal · script file append · hash change on script

Stage 9 — Beacon Pattern Replication

Second beacon_emitter_cloak pass replicating exfiltration beacon shape. DNS subdomain encoding, jittered HTTP callbacks. No data movement — beacon pattern only.

Validation

[VALIDATION]  apt_kill_chain
[RUN_ID]      155aa94b-7cf5-4ffd-a7e4-eddea1b3f8db
[EXPECTED]    25
[OBSERVED]    25
[PARTIAL]     0
[MISSING]     0
[COVERAGE]    100.0%
[SAFETY FAIL] 0
[VERDICT]     PASS

25 detection expectations. 25 matched. 0 missing. 0 safety violations. 339 synthetic telemetry events.

Full detection signal map:

Stealth Scoring

Model

Each layer receives a score (0–100) based on detection surface. Lower score = more detectable.

base_score = 100

penalties:
  3.0 per unique detection_opportunity signal (artifact-level)
  4.0 per alert_signature (manifest)
  2.0 per expected_event (manifest)
  2.5 per log source exposed

  20.0 if executable = True
  10.0 if files_created = True

bonus:
  +5.0 (one-time) if all artifacts are simulation_only + non-executable

deduplication: signals appearing in both artifact telemetry and manifest
are counted once — rewarding layers whose manifest accurately predicts
runtime behavior.

Example: mirror_loop_deflector (score: 84)

base_score          = 100
detection_ops (3)   → -9.0
alert_sigs (3)      → -12.0
expected_events (3) → -6.0
log_sources (3)     → -7.5
dedup credit        → +13.5  (overlap between manifest and runtime signals)
simulation bonus    → +5.0
                      ──────
                        84

Results

mirror_loop_deflector     ████████████████░░░░  84
beacon_emitter_cloak      ████████████████░░░░  80
lateral_webcrawler        ███████████████░░░░░  78
anti_forensics_molt       ███████████████░░░░░  77
memory_hijack_inheritor   █████████████░░░░░░░  68
poltergeist_file_infector █████████████░░░░░░░  68
dormant_sleeper_seed      █████████████░░░░░░░  65
shadow_system_rebuilder   █████████████░░░░░░░  65

C2 and evasion layers score highest — designed to blend. Persistence layers score lowest — systemd unit creation and file integrity violations are hard to hide. Memory injection sits in the middle: detectable, but requires deeper visibility than log analysis alone.

What to Do With the Output

339 synthetic telemetry events, JSONL format:

{
  "artifact_id": "c76c6616-764b-4e...",
  "layer": "memory_hijack_inheritor",
  "phase": "memory_probe",
  "behavior_class": "proc_mem_access",
  "mitre_techniques": ["T1055", "T1134"],
  "detection_opportunities": ["proc_mem_access", "rwx_region_write"],
  "simulation_only": true,
  "executable": false
}

• SIEM rule validation — feed the JSONL in and check which rules fire at which stage
• Sigma rule testing — detection signals map directly to Sigma condition fields
• Detection model training — synthetic labeled data for behavioral classifiers
• SOC tabletop exercises — replay the timeline and test analyst response
• Coverage gap analysis — compare [MISSING] expectations against your current rule set

The Repo

github.com/GnomeMan4201/shenron

git clone https://github.com/GnomeMan4201/shenron
cd shenron
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python3 shenron.py --scenario apt_kill_chain
python3 shenron.py --validate latest

Five built-in scenarios. Fifty behavioral simulation layers. All inert. Detection coverage scoring included.

Observable adversarial behavior, not portable adversarial procedure.

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

I was building Maatru, a Telugu literacy app for kids whose parents can’t comfortably teach the script. The original design had photo-feedback as the core interaction. A kid writes a letter on paper, takes a picture, the app compares the writing to the target and gives feedback. The mechanic felt right. Paper-and-pencil is how kids actually learn handwriting. And Gemma 4’s multimodal capability was one of the strongest things in the announcement.

Before I built anything, I tested whether the capability held up for my use case. It didn’t. Gemma 4 confidently misread typed Telugu characters as completely different characters. These were clean Unicode glyphs on white backgrounds, the easy case. Across 20 test samples spanning four difficulty tiers, the cloud variant (Gemma 4 31B Dense) got 4 right. The local variant (Gemma 4 E4B) got 1.

The interesting part of this story isn’t that the capability failed. It’s that I knew within a day, not three weeks into the build. This essay is about the evaluation discipline that produced that knowledge. And why I think anyone planning to build a non-trivial product on Gemma 4 (or any model) should run capability gates before architectural commitment.

To be clear, this isn’t a Gemma 4 criticism. The model does many things well. The multilingual generation and function calling ended up powering Maatru’s curriculum content and agentic planner. What I learned is that model spec sheets are claims to be tested. The cost of testing those claims early is much smaller than the cost of building on a capability that doesn’t deliver.

How I structured the evaluation

A capability gate is an evaluation that answers one question. Does this model capability hold up for my use case, at a quality bar that makes my product viable? For Maatru’s photo-feedback, the question was concrete. Can Gemma 4 reliably identify Telugu characters from images that look like what a kid actually produces?

I built the eval around four difficulty tiers. The typed tier was the easy case: five Telugu characters in a clean Unicode font, black on white, no noise. The adult-handwriting tier was five samples I wrote by hand. The child-handwriting tier was samples produced by a kid I knew, with shaky strokes and uneven spacing. The similar-looking tier was characters that share visual structure, the Telugu confusables set that kids commonly mix up. These test whether the model actually distinguishes between similar shapes or just pattern-matches loosely. Five samples per tier, twenty total. Small enough to run in an afternoon, large enough to surface real failure patterns.

I ran the eval against both Gemma 4 variants I had access to. E4B running locally through Ollama on an M4 MacBook Air with 16GB of RAM, and 31B Dense through OpenRouter’s free tier. Testing both variants matters because Gemma 4 spec claims apply to the family. Specific variants may perform differently due to parameter count, distillation, or fine-tuning. My acceptance threshold, decided before running: typed+adult tiers ≥80% combined, child tier ≥60%. If the model couldn’t reliably read clean reference glyphs and adult handwriting, the kid-handwriting case was hopeless. Defining the threshold before I saw results was important. Otherwise it’s too easy to convince yourself that bad numbers are “actually pretty good.”

What the data showed

The scorecard came back well below threshold. Cloud variant: 1 of 5 typed, 1 of 5 adult handwriting, 0 of 5 child handwriting, 2 of 5 similar-looking. Total 4 of 20, or 20%. Local variant: 1 of 5 typed, 0 of 5 adult, 0 of 5 child, 0 of 5 similar. Total 1 of 20, or 5%. The threshold required 80% on typed+adult combined. The cloud variant gave me 20% on that combined tier.

The failure mode mattered more than the aggregate number. I’d expected child-handwriting to be hard. Kids’ strokes are messy, spacing is unpredictable, letter shapes drift from the canonical form. What I didn’t expect was that the typed reference tier would also fail. The clearest example: Gemma 4 confidently misread అ, a basic Telugu vowel rendered cleanly on a white background, as ౦ (a Telugu numeral) on the cloud variant, and as ని (a different consonant-vowel combination) on the local variant. Same input image, two completely different wrong outputs from two independent inference paths. And neither model hedged. Both committed to wrong answers with no caveats.

That confidence without hedging is what made the failure mode dangerous for the product I was planning. A model that says “I think this might be అ but I’m uncertain” is a model you can build around. You handle the uncertainty in the UX. A model that says “this is ని” when it’s actually అ ships a bug straight into a literacy tool for kids. Different wrong answers on different runs across independent inference paths tells you you’re looking at a model knowledge gap on a specific subdomain. Telugu script recognition in Gemma 4’s vision training distribution. Not a fixable engineering bug.

Ruling out the alternatives

Before accepting “the model can’t reliably do this” as the explanation, I worked through the alternatives. The first candidate was image encoding. Maybe the test images weren’t reaching the model correctly. I ruled this out because the cloud variant got 4 of 20 right, including 2 of the similar-looking pairs. That means the model was receiving and processing visual input correctly. An encoding bug would produce uniform failure across all 20 samples, not the spread of failures I actually saw.

The second candidate was API throttling or silent quality degradation. Maybe OpenRouter’s free tier downgrades model output under load. I ruled this out because cloud providers don’t quietly serve degraded outputs. They return errors. They don’t return lower-quality answers silently. Also, the local E4B variant runs entirely on-device with no external dependency, and it produced even worse results than cloud. If both an unthrottled local run and a potentially-throttled cloud run produce the same failure pattern, throttling isn’t the cause.

Cold-start effects were the third candidate. Maybe the model needed warm-up calls before producing reliable output. I’d added warm-up calls to the runner before the timed loop. Latencies stabilized after warm-up. But the accuracy problem persisted across all 20 samples. Cold-start affects latency, not accuracy.

What remained was the conclusion the data supported. Gemma 4’s vision capability does not currently read Telugu script reliably enough to be the foundation of a literacy product. This is a model training-data limitation, not a fixable engineering bug. Indic scripts are likely underrepresented in the visual training corpus relative to Latin scripts. That isn’t specific to Gemma 4. Most big models have the same gap with non-English visual recognition. But it does mean any product built on Gemma 4 vision for Indic scripts (or other underrepresented visual writing systems) inherits this limitation.

What I’d tell another developer

The discipline that made this story end in a one-day pivot instead of a three-week wasted build is what I think of as capability-gate-first engineering. Before you commit to an architecture that depends on a specific model capability working at a specific quality bar, build the smallest possible evaluation that tests that exact capability under conditions that approximate your actual use case. Set the threshold before you see results. Run it on day one, not week three. If the model passes, you’ve validated a key assumption. If it fails, you’ve saved yourself weeks of building on something that won’t hold up.

For developers planning to build with Gemma 4 on non-English use cases: don’t trust the spec sheet’s “multimodal” or “multilingual” claims as proxies for your specific language or script. Test the exact capability you need on the exact data shape you’ll see in production. For my use case that meant 20 Telugu samples across 4 difficulty tiers. For yours it might be 50 Arabic OCR samples, or 30 medical-image inferences, or 100 Hindi-handwriting captures. The shape of the eval depends on the use case. The discipline of running it before architectural commitment doesn’t. Maatru shipped because the day-one capability gate told me to redesign around what Gemma 4 actually does well, multilingual text generation and agentic reasoning, rather than what the announcement promised. Honestly, that one day of testing saved the project.

Bootstrap 5 comes with built-in floating labels, but the default inputs can feel a bit too tall for compact forms.

Here’s a cleaner version with a more balanced height and a simple card layout.

The HTML structure:

<link
  href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css"
  rel="stylesheet"
/>

<div class="container py-4">
  <div class="row justify-content-center">
    <div class="col-md-6">
      <div class="card bg-white shadow-sm">
        <div class="card-header h6 py-3 fw-bold">
          Register
        </div>

        <div class="card-body">
          <form method="#" autocomplete="off">
            <div class="row">
              <div class="col-md-6">
                <div class="form-floating mb-3">
                  <input type="text" class="form-control" id="first_name" name="first_name" placeholder="First name">
                  <label for="first_name">First name</label>
                </div>
              </div>

              <div class="col-md-6">
                <div class="form-floating mb-3">
                  <input type="text" class="form-control" id="last_name" name="last_name" placeholder="Last name">
                  <label for="last_name">Last name</label>
                </div>
              </div>
            </div>

            <div class="form-floating mb-3">
              <input type="email" class="form-control" id="email" name="email" placeholder="Email address">
              <label for="email">Email address</label>
            </div>

            <div class="form-floating mb-3">
              <input type="password" class="form-control" id="password" name="password" placeholder="Password">
              <label for="password">Password</label>
            </div>

            <div class="form-floating mb-3">
              <input type="password" class="form-control" id="password_confirmation" name="password_confirmation" placeholder="Confirm Password">
              <label for="password_confirmation">Confirm Password</label>
            </div>

            <button type="submit" class="btn btn-primary w-100">
              Register
            </button>
          </form>
        </div>
      </div>
    </div>
  </div>
</div>

We need just a little bit of custom CSS:

.form-floating > .form-control {
    height: 48px;
    min-height: 48px;
    padding: 1rem 0.75rem 0.25rem;
  }

  .form-floating > label {
    padding: 0.75rem;
  }

  .form-control:focus {
    box-shadow: none;
  }

And here’s our nice form! 😊

MCP servers expose tools to LLMs, but most configs grant tools broader permissions than they need, ship without auth, and leak prompt-injection surface in tool descriptions. This scanner finds it before your model does.

Most MCP servers I’ve audited in the last few months had the same three issues:

1. A shell or fs tool was scoped to the entire filesystem when the use case needed exactly one directory.
2. The transport ran without auth because the local-dev SSE config got promoted to prod.
3. Tool descriptions echoed verbatim into prompts with no sanitization — a perfect injection surface.

@hailbytes/mcp-security-scanner is what I wish I’d had on day one of building MCP servers. It’s a static + dynamic scanner for MCP configs and live endpoints that flags these patterns.

CLI

# Scan a local config
npx @hailbytes/mcp-security-scanner ./mcp-config.json

# Scan a live endpoint
npx @hailbytes/mcp-security-scanner https://my-mcp-server.example.com

# SARIF output + fail the build
npx @hailbytes/mcp-security-scanner ./config.json --output=sarif --exit-code

Programmatic

import { scan } from "@hailbytes/mcp-security-scanner";

const report = await scan({ configPath: "./mcp-config.json" });

if (!report.passed) {
  console.error(report.findings);
  process.exit(1);
}

What it checks

• Overprivileged tools — broader permissions than the declared function needs (filesystem scope, shell access, network egress)
• Missing or weak authentication — unauthenticated transports, missing token validation, plaintext secrets in config
• Prompt injection surface — tool descriptions and output paths that pass through to model context without sanitization
• Unsafe defaults — insecure transport defaults, verbose error exposure, CORS wildcards

The SARIF output drops straight into GitHub Code Scanning, so findings show up as alerts on PRs — same place your SAST results live.

npm install -g @hailbytes/mcp-security-scanner

Source: github.com/hailbytes/mcp-security-scanner — MIT licensed. Pairs nicely with @hailbytes/mcp-server-template if you want a scaffold that comes up secure by default.