SignSpeak

Real-Time Sign Language Translation with MediaPipe, Flutter, and Gemini Nano

Build with AI – 2026 | Noor

SignSpeak is a mobile application designed to bridge the communication gap for the Deaf and Hard-of-Hearing community. By combining on-device computer vision with an on-device large language model, it translates American Sign Language (ASL) gestures and custom hand signs into natural, complete English sentences in real time – entirely offline, with no data sent to any server. Its defining feature is a fully customizable vocabulary: new signs, names, or even full phrase shortcuts can be added by any user without retraining from scratch.

1. MediaPipe: The Digital Skeleton of Gesture Recognition

What is MediaPipe?

MediaPipe is an open-source framework developed by Google for building multimodal, on-device machine learning pipelines. It provides production-ready solutions for tasks such as face detection, pose estimation, and – most relevant here – hand landmark detection. MediaPipe is designed to run efficiently on mobile hardware without requiring a network connection, making it a natural fit for a privacy-first application.

In SignSpeak, the Hand Landmarker solution is used. It detects and tracks the 3D coordinates of 21 individual points (landmarks) on a human hand from a single camera frame. These points cover every joint of every finger plus the wrist, providing a rich geometric representation of any hand shape or pose.

The 21 Hand Landmarks

Each landmark is expressed as a normalised (x, y, z) coordinate. The x and y values are pixel positions relative to the image width and height, and z represents depth relative to the wrist. The 21 points are distributed as follows:

Landmark 0 (WRIST) is the anchor for all normalisation. Landmark 9 (MIDDLE MCP) defines the span.

How MediaPipe is Used in SignSpeak

Since sign language frequently involves both hands, SignSpeak configures MediaPipe to track up to two hands simultaneously. For each detected hand, the (x, y, z) values of all 21 landmarks are extracted, producing a raw vector of 63 values per hand. With two hands, this gives 126 raw values per frame.

To make the model robust to differences in hand size, camera distance, and position within the frame, these coordinates are normalised before being used as model input:

• Wrist-relative: Each coordinate is shifted by subtracting the wrist position (landmark 0), so the wrist is always at the origin (0, 0, 0).
• Scale-normalised: Each shifted coordinate is divided by the Euclidean distance between the wrist and the middle finger MCP joint (landmark 9). This “span” acts as a scale factor, so the same sign made close to or far from the camera produces an identical feature vector.

The formula in Python is span = np.linalg.norm(mid_mcp - wrist). In Dart (Flutter), the equivalent must use sqrt(dx*dx + dy*dy + dz*dz) – not the squared distance – to match the training data exactly.

⚠️ Critical: Feature Extraction Must Match Exactly
The single most common cause of poor accuracy when moving from Python to Flutter is a mismatch in normalisation. The Python training pipeline uses np.linalg.norm (Euclidean distance, i.e. square root). The Flutter inference code must use the identical formula. Using squared distance instead will produce a completely different scale and render the model unreliable.

2. Gemini Nano: Bringing Intelligence to the Edge

What is Gemini Nano?

Gemini Nano is the smallest and most efficient model in Google’s Gemini family, purpose-built for on-device inference. Unlike cloud-based LLMs, Gemini Nano runs entirely on the device’s hardware using Android’s AICore runtime, requiring no internet connection and sending no user data off-device. It is available on supported devices including the Pixel 8 series and Samsung Galaxy S24 series.

Why an LLM for Sign Language?

ASL has a grammatically different structure from spoken English. Signers routinely omit articles (a, the), auxiliary verbs (is, are, was), and tense markers. A direct transcription of detected signs would produce telegraphic output such as “WATER NEED” or “NAME MY”. Gemini Nano’s role in SignSpeak is to act as an interpreter: it receives the raw sign tokens and a context-aware prompt, and returns a fluent English sentence (because LLMs are only next word predictors am I right :D). The prompt is designed with few-shot examples:

You are a sign language interpreter. Convert ASL sign tokens into
natural fluent English sentences.

WATER NEED       → I need some water please.
NAME MY NOOR     → My name is Noor.
HELP ME PLEASE   → Could you please help me?

Output ONLY the final sentence. No explanation. Under 15 words.

How to Install and Enable Gemini Nano on Android

Gemini Nano is accessed via the AICore system service. Enabling it requires the following steps on a supported device:

1. Update AICore: Open the Google Play Store and search for “AICore”. Ensure it is updated to the latest version.
2. Enable Developer Options: Go to Settings → About Phone and tap Build Number seven times until Developer Mode is enabled.
3. Enable Gemini Nano: In Settings → System → Developer Options, scroll to find “Gemini Nano” or search for “AICore Settings”. Toggle on “Enable Gemini Nano” and “Enable On-Device Model”.
4. Wait for model download: The on-device model downloads silently in the background. This can take 10–15 minutes on Wi-Fi. The device must be charging and connected to Wi-Fi. The model is not available until this download completes.
5. Verify availability: Connect the device via USB and run:

adb shell cmd aicore status

The output should confirm the model is downloaded and AICore is running. In SignSpeak, Gemini Nano is accessed via a Flutter MethodChannel that bridges to the Android-native AICore SDK. If the on-device model is unavailable, the app transparently falls back to the Gemini Flash API (cloud) or a rule-based sentence assembler.

3. The Python Pipeline: From Camera to Trained Model

The gesture recognition model is built using a three-stage Python pipeline located in the custom_model/ directory. All three scripts share the same MediaPipe feature extraction logic to ensure consistency between training data and inference.

collect.py – The Data Gatherer

Purpose

collect.py uses your PC webcam and MediaPipe to capture hand landmark data for each sign you want to recognise. Instead of recording raw video or images, it records only the 126 normalised landmark coordinates per frame – this is both compact and already in the exact format the model needs.

How to Use

Install dependencies:

pip install mediapipe opencv-python numpy

Run the script from the custom_model/ directory:

python collect.py

The webcam window opens showing a live hand skeleton overlay. For each sign in the SIGNS list:

1. Get into position with your hand(s) in the correct gesture.
2. Press SPACE. A 5-second countdown begins – use this time to settle into the pose.
3. The script automatically captures 10 frames at 0.5-second intervals. A white border flashes on each capture. The terminal prints each shot as OK or MISSED.
4. When you have enough samples for a sign, press N to advance to the next one.
5. Press Q at any time to quit and save all collected data so far.

ℹ️ Two-handed signs: Signs marked as TWO_HANDED (e.g. WELCOME, HELP, MORE) display a purple [TWO-HANDED] label. The “Hands:” counter in the top-right turns green only when the required number of hands are detected. For two-handed signs, ensure both hands are fully visible in the frame before pressing SPACE.

Customising the Sign Vocabulary

The list of signs to collect is defined at the top of the script:

SIGNS = [
    'none',       # always required as the negative class
    'hello',
    'my',
    'name',
    'noor',       # a custom sign - any gesture you define
    'welcome',
    # add more here...
]

'none' must always be first. It is the negative class – the gesture the model predicts when no recognisable sign is being made. Every other sign can be any word, name, or phrase you choose. The gesture itself is entirely up to you: pick any distinct, static hand shape for each token.

Adding New Signs Without Recollecting Everything

One of the most practical features of this pipeline is the ability to add new signs incrementally. Because the training data is stored as a plain CSV file, you can append rows for new signs without discarding the data you have already collected.

To add a new sign to an existing dataset:

1. Add the new sign to the SIGNS list in collect.py. Place it at the end, after all existing signs. Do not reorder or remove any existing entries – the label index (position in the list) must stay the same.
2. Set the starting index: At the bottom of the script, set sign_idx to the index of your new sign so collection jumps straight to it without re-running existing signs.
3. Run collect.py: New rows will be appended to the existing data/landmarks.csv. All previous sign data is preserved.
4. Update labels.txt: The script automatically rewrites data/labels.txt with the full updated list.
5. Retrain: Run python train.py. It reads the updated CSV, detects all present classes, and trains a new model with the expanded vocabulary.

ℹ️ Why you must retrain even for one new sign: The output layer of the MLP is sized to the number of classes. Adding a new sign changes that count, so a new model must be exported. However, all the previously collected data remains valid and is automatically included – you only spend time collecting samples for the new sign.

Files Generated by collect.py

• data/landmarks.csv – one row per captured frame, containing a label index followed by 126 normalised landmark coordinates.
• data/labels.txt – the ordered list of sign names, one per line. The line number (zero-indexed) is the label ID used in the CSV.

train.py – The Brain Builder

Purpose

train.py reads the collected CSV, trains a Multi-Layer Perceptron (MLP) neural network using TensorFlow, and exports the result as a quantised TFLite model ready for mobile deployment.

How to Use

Install dependencies:

pip install tensorflow scikit-learn pandas matplotlib

Run:

python train.py

Training typically completes in 1–3 minutes. EarlyStopping monitors validation accuracy and halts training once the model stops improving, so you do not need to manually tune the number of epochs.

Model Architecture

The classifier is a four-layer MLP. The input is the 126-value normalised feature vector; the output is a softmax probability distribution over the number of sign classes:

Input:   126 features
         ↓
Dense 128  + BatchNorm + Dropout(0.3)
         ↓
Dense 64   + BatchNorm + Dropout(0.2)
         ↓
Dense 32
         ↓
Dense N    + Softmax         (N = number of sign classes)

BatchNorm stabilises training between layers. Dropout prevents the network from over-relying on any single feature, which matters because many signs share similar finger configurations. The final Softmax layer outputs a confidence percentage for each class.

Files Generated by train.py

• exported_model/gesture_classifier.tflite – the trained model, quantised to INT8 (~45 KB). This is the file loaded by Flutter.
• exported_model/gesture_labels.txt – the label list corresponding to the model’s output indices. Must be copied to Flutter alongside the .tflite file.
• exported_model/training_curves.png – accuracy and loss curves across epochs for both training and validation sets.
• exported_model/confusion_matrix.png – a grid showing true vs. predicted labels across the test set. Signs on the diagonal are correctly classified; off-diagonal entries reveal confusions between visually similar signs.

test_model.py – The Real-World Check

Purpose

test_model.py lets you verify the trained model against your own live webcam before deploying to a phone. It is the fastest way to catch issues with specific signs before going through the Flutter build cycle.

How to Use

Install dependencies:

pip install tflite-runtime opencv-python

Run:

python test_model.py

The script opens your webcam, runs hand landmark detection, and performs inference on every frame. The display shows:

• The predicted sign label in large text at the bottom of the frame, greyed out when confidence is below the 75% threshold.
• A confidence percentage next to the label.
• A ranked bar chart of the top 3 predictions on the right side – useful for diagnosing which signs are being confused with each other.
• A live FPS counter and hand count in the top bar.

Target: Aim for consistent confident predictions (≥85%) on all signs before moving the model to Flutter. If two signs are frequently confused, check the confusion matrix from train.py and collect additional samples for those specific classes.

4. Importing the Trained Model into Flutter

Once test_model.py confirms the model performs well, it can be integrated into the Flutter app in five steps.

Step 1: Copy Model Files into the Project

copy exported_modelgesture_classifier.tflite  signspeakassetsmodels
copy exported_modelgesture_labels.txt         signspeakassetsmodels

Step 2: Register Assets in pubspec.yaml

Both files must be declared in pubspec.yaml so the Flutter build system bundles them into the app:

flutter:
  uses-material-design: true
  assets:
    - assets/models/

Step 3: Add the tflite_flutter Dependency

dependencies:
  tflite_flutter: ^0.12.1

Then run flutter pub get.

Step 4: Load and Run the Model

The TfliteGestureClassifier service handles loading, feature extraction, and inference:

// Initialise once at app startup
_interpreter = await Interpreter.fromAsset(
  'assets/models/gesture_classifier.tflite',
  options: InterpreterOptions()..threads = 2,
);

final labelData = await rootBundle.loadString(
  'assets/models/gesture_labels.txt',
);
_labels = labelData.trim().split('n');

At inference time, the raw MediaPipe landmarks are normalised and passed to the interpreter:

// Build the 126-feature input vector
final input  = [extractFeatures(allHands).toList()];  // [[f0..f125]]
final output = List.generate(1, (_) => List.filled(_labels.length, 0.0));

_interpreter.run(input, output);

// Map index to label
final maxIdx = output[0].indexWhere((p) => p == output[0].reduce(max));
final label  = _labels[maxIdx];
final conf   = output[0][maxIdx];

Step 5: Normalise Identically to collect.py

The feature extraction in Dart must be byte-for-byte equivalent to the Python normalisation. The critical rule is to use the square root (Euclidean norm), not the squared distance:

// Dart - must match collect.py exactly
final dx   = lm[9].x - wristX;
final dy   = lm[9].y - wristY;
final dz   = lm[9].z - wristZ;
final span = sqrt(dx * dx + dy * dy + dz * dz) + 1e-6;  // dart:math sqrt

for (int i = 0; i < 21; i++) {
  out[offset + i*3 + 0] = (lm[i].x - wristX) / span;
  out[offset + i*3 + 1] = (lm[i].y - wristY) / span;
  out[offset + i*3 + 2] = (lm[i].z - wristZ) / span;
}

✅ The app automatically detects the model. If gesture_classifier.tflite is present in assets/models/, TfliteGestureClassifier loads it on startup and the app switches from the built-in rule-based classifier to the trained model. The debug console prints TFLite classifier loaded: N classes to confirm. If the file is absent, the app falls back to geometry rules silently.

SignSpeak – Build with AI 2026
MediaPipe · Gemini Nano · TensorFlow Lite · Flutter

Hello, I’m Shrijith Venkatramana. I’m building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

Here are six small examples that reveal two completely different language philosophies.

With the recent Bun port to Rust (from Zig) with LLM assistance is setting the interwebs ablaze, I felt going a bit deeper into how Rust and Zig differ in their approaches to solving concrete problems.

Most discussions about Rust and Zig become philosophical quickly.

But the philosophy itself is just the starting point.

A deeper understanding can come from how each philosophy shapes day to day engineering tasks such as:

• allocating memory
• handling errors
• sharing state
• building binaries
• interfacing with C
• structuring dependencies

I’ll walk you through a a few examples, after which, the two languages may start feeling fundamentally different.

If one were to summarize, Rust keeps asking:

“How can the compiler make this difficult to misuse?”

Zig keeps asking:

“How can the programmer see exactly what is happening?”

Let’s go through a few examples.

Example 1: Creating a Dynamic Array

Let’s start with something boring.

A growable list of integers.

Rust

fn main() {
    let mut numbers = Vec::new();

    numbers.push(1);
    numbers.push(2);
    numbers.push(3);
}

Very little ceremony.

You do not think about allocation yet.

You do not think about cleanup.

You do not think about allocator strategy.

Rust intentionally compresses those concerns away unless you need them.

Zig

const std = @import("std");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();

    const allocator = gpa.allocator();

    var numbers = std.ArrayList(i32).init(allocator);
    defer numbers.deinit();

    try numbers.append(1);
    try numbers.append(2);
    try numbers.append(3);
}

The first reaction many engineers have:

“Why so explicit?”

But after a while, the important thing becomes obvious:

• allocator choice is visible
• allocation lifetime is visible
• cleanup is visible
• failure is visible

The language forces resource awareness into the code itself.

That changes how systems evolve.

In Rust, memory management becomes something the compiler heavily assists with.

In Zig, memory management remains part of the engineering model.

That difference shows up everywhere later.

Example 2: Invalid References

This example makes Rust’s philosophy impossible to miss.

Rust

fn main() {
    let mut values = vec![1, 2, 3];

    let first = &values[0];

    values.push(4);

    println!("{}", first);
}

Rust rejects this.

Because push() may reallocate the vector internally, invalidating the reference.

The interesting part is not the compiler error.

The interesting part is what Rust optimizes for.

This category of bug is historically painful:

• crashes under load
• intermittent corruption
• impossible‑to‑reproduce failures
• security vulnerabilities

Rust treats these as unacceptable failure classes.

So the compiler intervenes aggressively.

Zig

Zig allows the same pattern.

const std = @import("std");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();
    const allocator = gpa.allocator();

    var values = std.ArrayList(i32).init(allocator);
    defer values.deinit();

    try values.append(1);
    try values.append(2);
    try values.append(3);

    const first = &values.items[0]; // reference to first element

    try values.append(4); // may reallocate, invalidating 'first'

    std.debug.print("{}n", .{first.*});
}

This code compiles and runs – but the reference first may become dangling after the second append. The program can crash or read garbage.

Zig’s philosophy differs.

Zig assumes:

The programmer should understand when memory relocation is possible.

That sounds dangerous.

And sometimes it is.

But there is another side.

Rust scales correctness mechanically.

Zig scales understanding culturally.

That is a very different engineering assumption.

Example 3: Error Handling

The two languages start feeling stylistically similar while still revealing very different priorities.

Rust

use std::fs;

fn read_file() -> Result<String, std::io::Error> {
    let content = fs::read_to_string("config.json")?;
    Ok(content)
}

The ? operator keeps propagation concise.

Errors stay explicit.

Control flow remains visible.

This is one of Rust’s best design choices.

Zig

const std = @import("std");

fn readFile(allocator: std.mem.Allocator) ![]u8 {
    const file = try std.fs.cwd().openFile("config.json", .{});
    defer file.close();

    return try file.readToEndAlloc(allocator, 1024 * 1024);
}

The important difference is subtle.

Zig keeps allocation visible even during convenience APIs.

That becomes surprisingly important in large systems.

Many production issues are really resource issues in disguise:

• allocation spikes
• hidden copies
• unexpected buffering
• allocator fragmentation
• ownership confusion

Zig keeps reminding you those things exist.

Rust tries harder to abstract them safely.

Example 4: Shared State Across Threads

This is where Rust becomes genuinely impressive.

Rust

use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let counter = Arc::new(Mutex::new(0));

    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        handles.push(thread::spawn(move || {
            let mut num = counter.lock().unwrap();
            *num += 1;
        }));
    }

    for handle in handles {
        handle.join().unwrap();
    }
}

There is ceremony here.

But Rust does something powerful:

• ownership is tracked
• mutation is controlled
• thread safety is enforced
• races become much harder accidentally

Rust’s concurrency story is restrictive because unrestricted concurrency has historically been catastrophic.

The language deeply distrusts shared mutable state.

For good reason.

Zig

Zig provides the same primitives with less ceremony.

const std = @import("std");

var counter: i32 = 0;
var mutex = std.Thread.Mutex{};

fn worker() void {
    mutex.lock();
    defer mutex.unlock();
    counter += 1;
}

pub fn main() !void {
    var threads: [10]std.Thread = undefined;
    for (&threads) |*t| {
        t.* = try std.Thread.spawn(.{}, worker, .{});
    }
    for (threads) |t| {
        t.join();
    }
    std.debug.print("counter: {}n", .{counter});
}

Zig’s approach feels much closer to C.

The primitives are straightforward.

The language trusts you more.

That simplicity appeals – especially when debugging.

But correctness depends more directly on engineer discipline.

Rust tries to make dangerous concurrency difficult.

Zig tries to make concurrency understandable.

Example 5: Interfacing With C

This is one of Zig’s most interesting strengths.

Zig

const c = @cImport({
    @cInclude("sqlite3.h");
});

That is basically it.

Zig’s C interoperability story feels unusually direct because the language positions itself close to C semantics.

Many systems engineers notice this immediately.

Teams with large existing C codebases find incremental migration realistic.

Rust

Rust has strong FFI support.

But it usually looks more like:

use std::ffi::{c_int, c_void};

extern "C" {
    fn sqlite3_open(filename: *const i8, ppDb: *mut *mut c_void) -> c_int;
}

Or it involves:

• bindgen
• unsafe blocks
• wrapper layers
• ownership translation

Rust tries to preserve safety guarantees across language boundaries.

Zig optimises for minimal friction.

Again:

• Rust prioritises correctness constraints
• Zig prioritises mechanical transparency

The same pattern keeps reappearing.

Example 6: Building a Static Binary

This example seems operationally boring until you have maintained deployment pipelines long enough.

Zig

zig build -Dtarget=x86_64-linux-musl

Done.

The first time many infra engineers try this, the reaction is:

“Wait, seriously?”

Zig’s toolchain story feels aggressively practical.

Almost as if someone tired of fighting build environments designed it.

Rust

Rust’s tooling is excellent.

Cargo is excellent.

But static cross‑compilation in Rust often introduces real‑world friction:

• linker configuration
• musl edge cases
• OpenSSL issues
• Dockerised builders
• target setup
• CI complexity

None of this is impossible.

But operationally, Zig often feels mechanically lighter.

That matters more than people think – especially for small infrastructure teams.

One Thing That Becomes Obvious After These Examples

Rust and Zig do not compete on syntax.

They compete on where engineering responsibility should live.

Rust says:

“The compiler should aggressively constrain dangerous behaviour.”

Zig says:

“The programmer should see the dangerous behaviour directly.”

That difference affects:

• APIs
• tooling
• concurrency
• debugging
• deployment
• dependency culture
• architecture style

Even the ecosystems reflect it.

Rust’s ecosystem is rich, abstracted, and ambitious.

Zig’s ecosystem is smaller, simpler, and operationally direct.

The Interesting Part Is How These Philosophies Fail

Rust failure mode:

“The system became architecturally complicated because the compiler demanded precision.”

Zig failure mode:

“The system stayed understandable, but correctness depended heavily on engineer discipline.”

Both are real costs.

Both become more visible as systems and teams grow.

What I Would Personally Optimise For

If multiple teams will maintain critical infrastructure for years:

I would strongly lean Rust.

The compile‑time guarantees compound in value organisationally.

If a small systems team deeply values operational simplicity and explicit control:

Zig becomes extremely attractive.

Especially for:

• tooling
• infrastructure utilities
• embedded systems
• lightweight services
• C migration layers

The deeper insight is this:

Rust optimises for preventing mistakes. Zig optimises for exposing reality.

Depending on the system, either property can be more valuable.

Practical Checklist

Before choosing either language, ask:

• Will many engineers touch this code?
• Is deployment simplicity a major concern?
• Is memory corruption a serious business risk?
• Is debugging or correctness the bigger pain today?
• Does the team already understand systems programming deeply?
• Will this system survive for many years?
• Is operational complexity already growing too quickly?
• Are hidden abstractions becoming a problem?

Most teams answer these emotionally.

The operational consequences arrive later.

Final Takeaway

After enough examples, the philosophical split becomes very hard to unsee.

Rust keeps moving complexity toward the compiler.

Zig keeps moving complexity toward the engineer.

Both choices carry costs.

The real question is:

“Which failure mode does this system tolerate better?”

Over to you…

If you have built serious systems in either language:

Which part became the actual long‑term cost?

What became painful once the codebase, deployment surface, and team started growing?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

Most AI app prototypes look cheap.

Then production happens.

A developer tests an LLM feature with 20 prompts, gets a few good responses, and assumes the cost is manageable. But production cost is not based on one prompt. It is based on:

input tokens
output tokens
requests per user
users per day
retry rate
tool calls
prompt caching
conversation history
model choice

That is where many teams get surprised.

The mistake is simple: they estimate the cost of a single API call instead of estimating the cost of the full production workflow.

The basic LLM cost formula

At the simplest level, LLM cost is:

Input cost  = input tokens  × input price
Output cost = output tokens × output price

Total cost = input cost + output cost

But real applications are rarely that simple.

A chat application may include:

system prompt
user message
conversation history
retrieved context
tool schemas
intermediate reasoning steps
final response

So the real token budget looks more like:

Total input tokens =
  system prompt tokens
+ user prompt tokens
+ conversation history tokens
+ retrieved context tokens
+ tool schema tokens
+ intermediate step tokens

And output tokens may include:

assistant response
tool call arguments
intermediate responses
summaries
structured JSON outputs

That is why cost grows faster than expected.

Example: a simple AI assistant

Assume we are building an internal AI assistant.

Each request uses:

Input tokens per request: 3,000
Output tokens per request: 800
Requests per day: 10,000

Daily token volume:

Input tokens/day  = 3,000 × 10,000 = 30,000,000
Output tokens/day =   800 × 10,000 =  8,000,000

Monthly token volume:

Input tokens/month  = 900,000,000
Output tokens/month = 240,000,000

Now multiply that by your provider’s model pricing.

This is where architecture decisions start to matter.

A larger model may give better results, but if the workload is high-volume, the cost difference can become significant. A smaller model may be good enough for classification, routing, extraction, or summarization tasks.

The hidden cost: output tokens

Developers often focus on input tokens.

That is a mistake.

Output tokens are usually more expensive than input tokens for many LLM providers. Also, verbose responses increase latency and cost.

For example, this output style:

{
  "answer": "...",
  "reasoning": "...",
  "sources": [...],
  "confidence": "...",
  "follow_up_questions": [...]
}

may be useful, but it costs more than:

{
  "answer": "...",
  "sources": [...]
}

Structured output is great, but every field has a cost.

For production systems, response design is architecture.

RAG makes the cost calculation harder

In a Retrieval-Augmented Generation system, every request may include retrieved chunks.

Example:

System prompt: 800 tokens
User question: 100 tokens
Retrieved chunks: 5 chunks × 700 tokens = 3,500 tokens
Conversation history: 1,500 tokens
Output reserve: 800 tokens

Total request size:

text 800 + 100 + 3,500 + 1,500 = 5,900 input tokens

A RAG system is not just a vector database problem. It is also a token budgeting problem.

Agentic workflows multiply cost

Agentic AI makes cost estimation even more important.

A simple chat request may be one LLM call.

An agent task may involve:

1. Intent classification
2. Planning
3. Tool selection
4. Tool execution
5. Observation
6. Retry or correction
7. Final answer generation

That means one user request can become 5 to 10 LLM calls.

If each step uses different prompts, context, and outputs, the real cost is:

Cost per user task =
  planning cost
+ tool-selection cost
+ tool-call argument generation cost
+ observation summarization cost
+ retry cost
+ final answer cost

This is why “cost per API call” is the wrong metric for agents.

The better metric is: cost per completed task

Prompt caching can change the economics

If your provider supports prompt caching, repeated prompt sections can be cheaper.

Good candidates for caching:

system prompts
policy instructions
tool definitions
schema descriptions
static business rules
large reusable context blocks

But caching only helps when the repeated portion is stable.

Bad candidates for caching:

user-specific data
frequently changing retrieved chunks
dynamic conversation history
real-time tool outputs

So, when estimating cost, separate your input into:

cacheable tokens
non-cacheable tokens

This gives a more realistic estimate.

What architects should estimate before production

Before shipping an LLM feature, estimate at least these numbers:

Average input tokens per request
Average output tokens per request
Requests per active user per day
Expected daily active users
Retry rate
Cache hit percentage
Number of LLM calls per workflow
Peak traffic multiplier
Monthly token volume
Cost per user
Cost per business transaction

For enterprise systems, also estimate:

cost by use case
cost by department
cost by model
cost by workflow step
cost by failed request
cost with and without caching

This helps prevent surprises later.

A practical example

Suppose we have:

Daily active users: 2,000
Requests per user per day: 15
Input tokens per request: 4,000
Output tokens per request: 700
LLM calls per workflow: 2
Retry rate: 10%

Total daily workflows:

2,000 × 15 = 30,000 workflows/day

Including two LLM calls per workflow:

30,000 × 2 = 60,000 LLM calls/day

Including 10% retry rate:

60,000 × 1.10 = 66,000 effective calls/day

Daily input tokens:

66,000 × 4,000 = 264,000,000 input tokens/day

Daily output tokens:

66,000 × 700 = 46,200,000 output tokens/day

Monthly estimate:

Input tokens/month  = 7.92 billion
Output tokens/month = 1.386 billion

Now the architecture conversation changes.

This is no longer a “small AI feature.” This is a production inference workload.

Ways to reduce LLM cost

The most effective cost controls are usually architectural, not prompt hacks.

1. Use smaller models for simple steps

Do not use a frontier model for every task.
Use smaller or cheaper models for:

classification
routing
metadata extraction
summarization
guardrail checks
format validation

Use larger models only where reasoning quality matters.

1. Reduce unnecessary context

More context is not always better.

In RAG systems, sending 15 chunks when 5 are enough increases cost and can reduce answer quality.

Control:

chunk count
chunk size
conversation history
tool schema size
metadata verbosity

1. Cache stable prompt sections

If the same system prompt, policy text, or tool definitions are reused across requests, caching can reduce cost.

Design prompts with reusable stable sections.

1. Summarize long conversations

Instead of sending the full conversation every time, maintain a compact memory summary.

Example:

Full conversation history: 12,000 tokens
Compressed summary: 1,200 tokens

That difference matters at scale.

1. Measure cost per workflow, not cost per call

A single user action may trigger multiple model calls.

Track:

cost per chat response
cost per document processed
cost per support ticket resolved
cost per SQL answer generated
cost per agent task completed

Business-level cost is more useful than API-level cost.

The real lesson

LLM cost estimation is not just finance work. It is architecture work.

Your cost depends on:

model choice
prompt design
context size
RAG strategy
agent loop design
tool schema size
retry handling
caching strategy
output format

That means developers and architects should estimate cost before shipping, not after the bill arrives.

I created a free calculator for this:

👉 LLM Inference Cost Calculator

It helps estimate:

daily cost
monthly cost
cost per user
cost per workflow
token volume
prompt caching impact
multi-model comparison

You can also explore the broader calculator hub here:
👉 SuperML AI Calculators

Final thought

The future of AI architecture will not be only about choosing the best model.

It will be about choosing the right model, for the right step, with the right context, at the right cost.

That is the difference between a demo and a production AI system.

LLM Inference Cost Calculator — Estimate Production LLM API Costs | SuperML — SuperML.dev

Estimate daily and monthly LLM inference costs across OpenAI, Anthropic, Google, Mistral, and Meta models. Includes prompt caching, cost-per-user, token burn-down, and multi-model comparison.

superml.dev

This is a submission for the Hermes Agent Challenge

The most expensive thing most AI agents forget is not your name. It’s the work they just did.

You let an agent spend thousands of tokens learning your environment, inspecting your repository, debugging issues, figuring out project conventions, and working through a useful engineering task—sometimes even discovering a workflow that reliably works. Then you close the session, come back the next day, and much of that context is gone. The repository is still there, but the agent has forgotten what it learned about it, how it solved the problem, and which workflow actually got the job done.

For short-lived tasks, that’s not a huge issue. If all you need is a summary, a SQL query, or a quick browser automation task, stateless execution works fine. But once agents start touching repeated engineering work, automation, or operational workflows, forcing them to rediscover the same solutions over and over becomes an expensive design flaw.

That’s what makes Hermes Agent from Nous Research interesting.

Hermes is not being pitched as just another coding copilot or a chatbot wrapper with tool access bolted on. The more ambitious idea is that successful execution should create reusable operational knowledge. If an agent solves a meaningful problem once, it should not have to relearn the same workflow from scratch the next time.

If that works reliably, it changes what open-source agents can become.

The Stateless Agent Problem

Most current agent frameworks effectively behave like this:

Goal → Plan → Tool Calls → Execute → Return Result → Forget

That lifecycle works surprisingly well—until the work becomes repetitive.

Imagine asking an agent to scan a repository, identify missing license headers, generate patches, run tests, and commit validated changes. A capable system might spend significant time inspecting the filesystem, inferring project conventions, handling failures, and refining its approach before it gets the task right.

Now run that exact same task a week later. Most agents will start from zero as though the previous execution never happened.

The same thing happens with recurring operational issues. If an agent spends twenty minutes discovering that a flaky CI failure came from one dependency mismatch and a bad environment variable, you would reasonably expect that discovery to be reusable. Instead, most systems replay the entire debugging process.

That’s the inefficiency. A human engineer would either remember the pattern or document the solution. Stateless agents generally do neither.

What Hermes Is Trying to Change

Hermes attempts to change that lifecycle by inserting a learning loop. Instead of behaving like a linear sequence, the intended model looks more like this:

Observe → Plan → Execute → Evaluate → Crystallize Skill → Reuse

The important difference is what happens after execution. Rather than treating task completion as the end of the interaction, Hermes evaluates whether the workflow it just used is worth keeping.

• Did the task succeed?
• Which actions actually mattered?
• Was the solution a one-off workaround, or does it represent a reusable operational pattern?

If the answer is yes, Hermes can retain that workflow as a reusable Skill instead of forcing the model to rediscover the same process later. That’s a much more compelling idea than simply preserving chat history.

What a Skill Actually Looks Like

“Procedural memory” sounds abstract until you think about what is actually being stored. Hermes’ approach to procedural workflows is much closer to inspectable skill artifacts than opaque memory blobs, which is a much healthier model than treating memory as hidden vendor state.

Conceptually, a crystallized skill artifact looks something like this:

# repo-license-remediation
version: 1.2

tags:
  - python
  - repository
  - compliance

inputs:
  - repo_path
  - license_header

required_tools:
  - filesystem
  - regex_match
  - file_write
  - terminal
  - git

steps:
  1. Scan Python files
  2. Detect missing headers
  3. Generate patch
  4. Run tests
  5. Commit validated changes

This is fundamentally different from remembering prior prompts or conversation snippets. This is operational knowledge.

Remembering that I prefer concise responses is personalization. Remembering how to safely repair a repository issue is actual capability. That distinction matters.

Procedural Memory Is More Interesting Than Chat Memory

A lot of AI products advertise memory, but most of the time that means conversation continuity, user preferences, or retained prompt context. That is useful, but it does not necessarily make the system better at doing work.

The more meaningful distinction is between remembering facts and remembering procedures. Humans become effective engineers because they internalize repeatable workflows. You do not memorize every exact command forever, but you do remember how to approach a recurring integration issue or how to remediate a familiar failure pattern.

Hermes is aiming much closer to that model. Its architecture can be thought of in three distinct layers:

• Working Memory: Short-lived execution state including the current task context, temporary variables, and recent tool outputs. This is standard agent behavior.
• Episodic Memory: Longer-lived contextual recall mapping project metadata, user preferences, and prior historical decisions to improve continuity.
• Procedural Memory: The interesting layer. It stores reusable workflows like debugging routines, deployment procedures, remediation pipelines, and integration playbooks.

If this layer works well, the system improves with repetition instead of simply remembering conversations.

The Scaling Problem

Persistent procedural memory sounds great until you hit the obvious question: What happens when the agent accumulates hundreds of workflows? Dumping all of them into the context window every time would be terrible for token efficiency, latency, and reasoning quality. A staged retrieval model makes much more sense:

• Discovery Stub (~20 tokens): Start with the minimum—just the skill name and a short description to determine relevance.

  Example: Python repository license remediation workflow

• Signature Layer (~200 tokens): If the workflow looks useful, retrieve expected inputs, required tools, and configuration assumptions to validate applicability.

• Blueprint Layer (~1,000+ tokens): Only when the workflow is actually executed do you load the full steps, command sequences, and tool invocation logic.

This is dramatically more scalable than brute-force memory stuffing. One caveat: if you are evaluating Hermes critically, it is worth checking which parts of this are implemented exactly as described today versus which represent broader architectural direction. But conceptually, this is the right shape of solution.

Trying Hermes Yourself

Setup itself looks straightforward, but the more interesting question is not whether Hermes can run—it’s whether repeated tasks actually become smarter.

The baseline setup follows a quick terminal sequence:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
hermes

Hermes also supports running beyond the terminal—across messaging interfaces and isolated execution backends—which makes the architecture feel operational rather than purely conversational.

A worthwhile test is giving it something that actually requires multi-step reasoning, like scanning a test repository, identifying missing headers, and generating a local patch. The real validation is whether the second execution feels noticeably cleaner, faster, and less wasteful.

Where Hermes Fits Compared to Other Frameworks

• LangChain: Gives developers raw building blocks and enormous flexibility. That is great if you want full architectural control, but it also means assembling everything yourself. Hermes feels more opinionated out of the box.
• AutoGen: AutoGen shines in multi-agent conversational workflows, but conversation-heavy systems can become noisy and expensive fast. Hermes feels less focused on agent dialogue and more focused on raw execution workflows.
• OpenDevin: Clearly aligned with software engineering automation and workspace environments. Hermes feels slightly broader, aiming at general operational agent behavior.
• OpenClaw: Adjacent rather than directly competitive. OpenClaw is strong around orchestration and communication routing, while Hermes is more interesting around procedural learning and self-improving execution.

Infrastructure Design Matters

A powerful agent without execution isolation is a liability. Giving unrestricted shell access to an LLM is not a serious production strategy.

Hermes supports multiple execution backends, including restricted local execution, Docker containers, SSH environments, Singularity, and remote sandbox environments like Modal. This matters because practical automation requires isolation.

A realistic workflow might involve a Slack alert triggering an isolated environment, Hermes validating deployment state, detecting a known failure pattern, applying a learned remediation workflow, and reporting back. That starts looking much less like a toy demo and much more like something operations teams could actually use.

Where This Can Go Wrong

The risks here are real:

• Skill Drift: A workflow that worked six weeks ago may be wrong today because dependencies changed, APIs evolved, or CLI flags broke. Without revalidation, procedural memory becomes stale automation debt.
• Faulty Generalization: An agent might incorrectly promote a brittle edge-case fix into a reusable standard workflow. That becomes dangerous quickly because repeated incorrect automation is often worse than forcing fresh reasoning every time.
• Security Risk: Persistent procedural memory can preserve unsafe commands, environment assumptions, or patterns that risk credential leakage. Any self-improving execution system needs strict governance.

Practical Safeguards

If anyone plans to use something like this in production, the minimum checklist probably includes:

• Human approval before newly created skills are allowed into active reuse.
• Verification across multiple successful runs before allowing autonomous reuse.
• Automated smoke testing, version control, and strict least-privilege execution boundaries.

Without those controls, procedural memory becomes accumulated operational risk instead of useful automation.

Why This Matters for Open Source

The bigger issue here is ownership.

A lot of proprietary AI systems assume persistent memory belongs inside vendor infrastructure. That creates lock-in, opaque automation logic, poor auditability, and painful migration stories.

If an open agent can retain reusable operational knowledge in inspectable, version-controlled artifacts, teams can audit agent behavior, share playbooks, migrate freely, and actually own the workflows the system learns. If your agent discovers something useful, that knowledge should live in infrastructure you control—not disappear into a hosted memory layer.

That may be the more important shift.

Discussion

1. Would you trust a self-improving agent for non-critical automation today? Why or why not?
2. What specific safeguards would you require before letting one touch production infrastructure?