Mert

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

Table of Contents

1. What We Built
2. Demo
3. Code
4. How We Used Gemma 4
5. The Genie Team

What We Built

Most AI assistants are text boxes with a microphone icon. You speak. Your words leave your device. A server somewhere thinks. A server responds. If you’re on a plane, in a rural clinic, or behind a firewall — nothing works.

We built the opposite.

Genie is an autonomous AI agent that runs Gemma 4 directly on your Android GPU through Google’s LiteRT-LM SDK. It doesn’t call an API. It doesn’t stream to a cloud. It sees your screen, controls your apps, reads your documents, remembers your preferences, teaches you concepts on a whiteboard, and executes multi-step tasks — all on-device, all offline.

The Problem: Running an LLM Is Easy. Letting It Touch Your OS Is Not.

Google gave us a great on-device inference SDK. Loading a Gemma model, creating a conversation, streaming tokens — that’s a few method calls. The hard part is everything around it.

When your agent can tap buttons, type into fields, and open apps autonomously, you have a fundamentally different safety problem than a chatbot. A chatbot that hallucinates gives you wrong text. An agent that hallucinates taps “Confirm Payment” on your PayPal screen.

That constraint shaped every architectural decision in Genie.

System Architecture Overview

Here’s the full system at a glance:

Wake Word (Vosk) → STT (Android) → AgentOrchestrator
    ↓
PromptBuilder → Planner → GenieEngine (LiteRT-LM / Gemma 4)
    ↓
Decision.Act → RiskAssessor → [Biometric HITL?] → ToolRegistry → OS Action
    ↓
ToolOutcome → History → SlidingWindow → Next Planning Turn
    ↓
Decision.Finish → Skill Cache Write → TTS Response → Resume Wake Word

Layer 1: The Voice Pipeline — Two Engines, One Microphone

Most voice assistants use a single speech engine. We use two, and the reason is physics.

Vosk is a lightweight, offline speech recognition library. We run it continuously in the background at 16kHz, listening for exactly one word: “Gemma”. It uses ~30MB RAM and has near-zero latency for wake-word detection.

Android SpeechRecognizer is heavier but far more accurate for full sentences. It only activates after Vosk detects the wake word.

// Vosk wake-word detection
override fun onPartialResult(hypothesis: String?) {
    val json = JSONObject(hypothesis)
    val partial = json.optString("partial", "")
    if (partial.lowercase().contains("gemma")) {
        speechService?.stop()           // Kill Vosk
        setUiState(AgentUIState.Waking)  // Show overlay
        startSttListening()              // Start full STT
    }
}

// STT captures the actual command
override fun onResults(results: Bundle?) {
    val text = results?.getStringArrayList(
        SpeechRecognizer.RESULTS_RECOGNITION
    )?.getOrNull(0) ?: ""
    dispatchToAgent(text)  // → AgentOrchestrator
}

Why not just use SpeechRecognizer for everything? Because it’s expensive. Running full-sentence recognition 24/7 drains battery and hogs the microphone. Vosk is purpose-built for always-on keyword spotting with minimal resource usage.

Layer 2: Intercepted Execution (The Inference Bridge)

The GenieEngine wraps Google’s LiteRT-LM SDK. The most important design decision lives in one line:

val newConversation = engine.createConversation(
    ConversationConfig(
        samplerConfig = agentSamplerConfig(),
        systemInstruction = PromptFormatting.buildSystemInstruction(systemPrompt),
        tools = tools,
        automaticToolCalling = false,  // ← This is everything
    )
)

Why automaticToolCalling = false?

In default mode, LiteRT-LM detects a tool call in the model’s output, executes it automatically, and feeds the result back — all without the application knowing. That’s fine for a chatbot generating weather data.

It’s catastrophic for an agent that can tap “Send $500” on your banking app.

By disabling automatic tool calling, every single tool call from the model passes through our code first. We intercept it. We validate it. We optionally require biometric authentication. Only then do we execute.

The Callback-to-Flow Bridge

LiteRT-LM uses a callback-based API (MessageCallback). We convert that into a Kotlin callbackFlow so the agent loop can consume responses asynchronously:

fun sendMessageAsync(contents: Contents): Flow<AgentResponse> = callbackFlow {
    conversation.sendMessageAsync(
        contents,
        object : MessageCallback {
            override fun onMessage(message: Message) {
                if (message.toolCalls.isNotEmpty()) {
                    trySend(AgentResponse.ToolCallRequest(message))
                } else {
                    trySend(AgentResponse.Token(message.toString()))
                }
            }
            override fun onDone() {
                trySend(AgentResponse.Done)
                close()
            }
            override fun onError(throwable: Throwable) {
                val error = if (throwable is CancellationException) {
                    ErrorTaxonomy.TransientErr("Inference cancelled", throwable)
                } else {
                    ErrorTaxonomy.FatalErr("Inference error: ${throwable.message}", throwable)
                }
                trySend(AgentResponse.Error(error))
                close()
            }
        }
    )
    awaitClose { }
}

Layer 3: The Agent Loop — Planning Like a Human

Here’s where Genie diverges from every other “AI assistant.” When you say “Open WhatsApp and send hi to Mom”, a chatbot tries to answer in one shot. Genie plans.

The AgentOrchestrator runs a loop that mirrors human problem-solving:

1. Observe — Read the screen
2. Plan — Decide the next action
3. Act — Execute one tool
4. Evaluate — Check the result
5. Repeat — Until the goal is done or a circuit breaker trips

The State Machine

data class AgentState(
    val goal: String,
    var intent: AgentIntent? = null,
    var plan: AgentPlan? = null,
    var currentStepIndex: Int = 0,
    val history: MutableList<HistoryEntry> = mutableListOf(),
    var retryCount: Int = 0,
    var replanCount: Int = 0,
    val maxRetries: Int = 3,
    val maxReplans: Int = 3,
    var isNovelPlan: Boolean = true,
)

Every tool call, every result, every error — all tracked in history. This history feeds back into the next planning prompt so the model always knows what it has already tried.

The Decision Type System

The planner produces exactly one of three decisions per turn:

sealed class Decision {
    data class Act(val tool: String, val args: Map<String, String>) : Decision()
    data class Finish(val summary: String) : Decision()
    data class Reply(val message: String) : Decision()
}

• Act → Execute a tool and continue the loop
• Reply → Speak to the user and stop
• Finish → Mark the goal complete

Plain text during planning is treated as invalid. The system prompt forces exactly one native tool call per turn:

“Call EXACTLY ONE tool per turn. No markdown, no extra text.”

The Sliding Window

The model has a limited context window. The SlidingWindowManager ensures the model always sees what matters:

• Keeps the first entry (the user’s goal — always visible)
• Keeps the last 9 entries (recent actions and results)
• Prunes transient errors after a success — if click("Wi-Fi") failed twice then succeeded, those two failures are removed from history

fun pruneAfterSuccess(history: MutableList<HistoryEntry>) {
    val lastEntry = history.lastOrNull() as? HistoryEntry.ToolResult ?: return
    if (lastEntry.outcome !is ToolOutcome.Ok) return

    val successToolName = lastEntry.toolName
    var index = history.size - 2
    while (index >= 0) {
        val entry = history[index]
        if (entry is HistoryEntry.ToolResult &&
            entry.toolName == successToolName &&
            entry.outcome is ToolOutcome.TransientErr) {
            history.removeAt(index)
        } else {
            break
        }
        index--
    }
}

Layer 4: The Safety Net — Dynamic Risk Assessment + Biometric HITL

Most AI safety systems use static per-tool flags: “this tool is dangerous, always ask for confirmation.” That’s crude. Opening Settings is safe. Opening PayPal and clicking “Send” is not — but both use the same click tool.

Genie’s RiskAssessor is dynamic. It evaluates the current screen context in real time:

object RiskAssessor {
    private val DESTRUCTIVE_VERBS = setOf(
        "send", "transfer", "pay", "confirm", "submit",
        "delete", "remove", "purchase", "authorize",
    )

    private val CURRENCY_REGEX = Regex(
        """[$€£¥₦₹₩₿]s*d+|d+.d{2}s*(USD|EUR|GBP|NGN)""",
        RegexOption.IGNORE_CASE,
    )

    fun assess(
        toolName: String,
        args: Map<String, String>,
        screen: ScreenContext
    ): RiskVerdict {
        if (toolName !in setOf("click", "tap_at", "type_text")) {
            return RiskVerdict.Allow
        }

        val signals = mutableListOf<RiskSignal>()

        if (isFinancialScreen(screen)) signals.add(FINANCIAL_SCREEN)
        if (isSensitiveApp(screen))    signals.add(SENSITIVE_APP)
        if (isDestructiveVerb(target)) signals.add(DESTRUCTIVE_VERB)
        if (isSensitiveField(screen))  signals.add(SENSITIVE_FIELD)
        if (isAuthFlow(screen))        signals.add(AUTH_FLOW)

        return if (signals.size >= 2) {
            RiskVerdict.RequireBiometric(reason)
        } else {
            RiskVerdict.Allow
        }
    }
}

The key insight: ≥2 independent signals required. A single signal (e.g., seeing a dollar sign) doesn’t trigger auth — that would cause false positives everywhere. But a currency symbol plus the word “Send” as a click target? That’s a real financial action. Biometric required.

The Biometric Bridge

When auth is required, HITLInterceptionWrapper launches a transparent activity that shows Android’s BiometricPrompt:

object HITLInterceptionWrapper {
    val authResultChannel = Channel<AuthResult>(capacity = 1)

    suspend fun executeWithAuth(
        tool: GenieTool,
        args: Map<String, String>,
        serviceContext: ToolServiceContext,
        appContext: Context,
        reason: String,
    ): ToolOutcome {
        appContext.startActivity(
            Intent(appContext, BiometricAuthActivity::class.java)
        )

        val result = withTimeoutOrNull(30_000L) {
            authResultChannel.receive()
        }

        return when (result) {
            is AuthResult.Approved -> tool.execute(args, serviceContext)
            is AuthResult.Denied  -> ToolOutcome.AuthErr("User denied")
            else                  -> ToolOutcome.AuthErr("Auth timed out")
        }
    }
}

The BiometricAuthActivity is invisible (Theme.Translucent.NoTitleBar), excluded from recents, and immediately finishes after sending the result.

Layer 5: Error Taxonomy — Four Tiers of Failure

Not all errors are equal. Genie classifies every failure into one of four tiers, each with its own recovery strategy:

sealed class ToolOutcome {
    data class Ok(val result: String) : ToolOutcome()
    data class TransientErr(val message: String) : ToolOutcome()
    data class LogicErr(val message: String) : ToolOutcome()
    data class AuthErr(val message: String) : ToolOutcome()
    data class FatalErr(val message: String) : ToolOutcome()
}

Circuit Breakers

Two circuit breakers prevent infinite loops:

1. Consecutive failure breaker: 5 consecutive failures of any kind → abort
2. Unknown tool breaker: Same non-existent tool requested 3 times → abort

if (consecutiveFailureCount >= 5) {
    Log.e(TAG, "Circuit breaker triggered")
    return "I ran into repeated errors. Please try again."
}

Layer 6: Self-Improvement — The Skill Cache

Every time Genie completes a novel task, it serializes the successful plan and stores it in a local Room database:

@Entity(tableName = "skills")
data class Skill(
    @PrimaryKey(autoGenerate = true) val id: Int = 0,
    val goalPattern: String,
    val planJson: String,
    val successCount: Int = 0,
    val createdAt: Long = System.currentTimeMillis(),
)

Next time you ask for something similar:

val skills = skillDao.findMatchingSkills("turn on wi-fi")
val bestSkill = skills.maxByOrNull { it.successCount }

If a match is found, Genie replays the cached plan step-by-step without invoking the LLM. No inference needed. Instant execution.

Every successful replay increments successCount. Over time, more reliable skills are prioritized. If a cached plan fails (e.g., UI drift from an app update), the agent falls back to live planning.

Layer 7: The Hands — 53 Tools Across 6 Families

Tools don’t touch the AccessibilityService directly. They go through the ToolServiceContext interface — a seam that lets us mock every OS action in tests:

interface ToolServiceContext {
    suspend fun clickElement(target: String): Boolean
    suspend fun typeText(text: String): Boolean
    suspend fun swipe(direction: String): Boolean
    suspend fun readScreen(): String
    suspend fun openApp(name: String): Boolean
    // ... 48 more
}

Tool Families

The Gesture System

Every gesture is a suspend function wrapping dispatchGesture() into a coroutine:

private suspend fun dispatchGesture(
    gesture: GestureDescription
): Boolean {
    return suspendCancellableCoroutine { continuation ->
        service.dispatchGesture(
            gesture,
            object : GestureResultCallback() {
                override fun onCompleted(desc: GestureDescription) {
                    if (continuation.isActive) continuation.resume(true)
                }
                override fun onCancelled(desc: GestureDescription) {
                    if (continuation.isActive) continuation.resume(false)
                }
            },
            null
        )
    }
}

The Profile System — 9 Modes, One Agent

Not every task needs full autonomous planning. Genie has 9 specialized profiles:

The key insight: not every feature needs agent autonomy. Scribe and Health have fixed, deterministic workflows. Adding an agent layer would only introduce latency and hallucination risk. Those profiles bypass the orchestrator entirely.

enum class ToolProfile(
    val id: String,
    val toolNames: Set<String>,
) {
    Chat(toolNames = setOf("reply", "save_fact", ...)),
    AppControl(toolNames = setOf("reply", "open_app", "click", ...)),
    Scribe(toolNames = emptySet()),  // UI-driven
    Health(toolNames = emptySet()),  // UI-driven
}

Memory System — Facts and Preferences

When a user says “Remember that I’m allergic to peanuts”, the agent calls save_fact(key="allergy", value="peanuts"). This is stored in Room:

@Entity(tableName = "user_facts")
data class UserFact(
    @PrimaryKey(autoGenerate = true) val id: Int = 0,
    val key: String,
    val value: String,
    val createdAt: Long = System.currentTimeMillis(),
    val updatedAt: Long = System.currentTimeMillis(),
)

Every fact is injected into every prompt:

## User Preferences
- allergy: peanuts
- favorite_restaurant: Mama Cass
- preferred_language: Yoruba

The Teaching Profile — An AI Tutor With a Whiteboard

Most AI tutors dump a wall of text. Genie teaches visually.

When you say “Teach me about photosynthesis”, the agent:

1. Creates a teaching board scene
2. Places a title card
3. Adds content cards with real definitions, formulas, and examples
4. Generates visualize_concept diagrams (flowcharts, timelines, mind maps)
5. Narrates each step with synchronized text-to-speech

Each step is one tool call. The user controls pacing — say “next” and the agent adds the next concept. The agent never ends the lesson on its own.

Observability — The Event Logger

Every significant event flows through an async event bus:

📦 Bootstrap: engine_init [3421ms]
⚡ State: idle → planning
🧠 Inference: 847ms, 156 tokens
✅ Tool: open_app({name=Settings}) [234ms]
✅ Tool: click({target=Wi-Fi}) [143ms]
⚡ State: executing → finished
📚 Skill written: 'turn on wi-fi' (3 steps)

Events are emitted with trySend() — non-blocking, never delays the agent loop. A dedicated coroutine consumes the Channel<GenieEvent> and writes to Logcat.

What We Learned

1. automaticToolCalling = false is non-negotiable for agents. If your model can touch the OS, you must intercept every tool call. No shortcut.
2. Two-signal risk thresholds prevent false positives. A single “Send” button doesn’t mean danger. “Send” on a financial screen does.
3. Not every feature needs agent autonomy. Fixed workflows are faster, more reliable, and more predictable with direct UI control.
4. Error history is a feature, not a log. Feeding errors back into the planning prompt lets the model self-correct. Pruning resolved errors keeps context clean.
5. Skill caching is the simplest form of self-improvement. No fine-tuning, no RLHF. Serialize what worked. Replay it next time.

Project Statistics

• 53 registered tools across 6 families
• 9 specialized profiles
• 4-tier error taxonomy with 2 circuit breakers
• 5-signal real-time risk assessor with biometric HITL
• Room-backed skill cache with success-count ranking
• 0 cloud dependencies
• 0 API keys

Demo

Watch Genie navigate Android, read files, control tools, and manage biometric security locks:

📲 Try it yourself: Download the APK from Google Drive

Code

The complete source code is fully open-source and modularized for Android:

🔗 GitHub Repository: github.com/Akeem1955/Genie

How We Used Gemma 4

Genie is entirely powered by local inference using the following models from Google’sallowlist:

1. litert-community/gemma-4-E2B-it-litert-lm (Effective 2-Bit Quantized) – 2.4 GB file size, 32K context window.
2. litert-community/gemma-4-E4B-it-litert-lm (Effective 4-Bit Quantized) – 3.4 GB file size, 32K context window.

Why Gemma 4 was the right fit:

• Resource Constrained Optimization: Running a large language model on consumer mobile devices demands aggressive resource control. Gemma 4 E2B is optimized to run at high speed (token generation rate) directly on the mobile GPU using the LiteRT-LM engine. It operates comfortably inside the Android service layer without causing system Out-Of-Memory (OOM) faults.
• Structured Tool Calls: Even with 2-bit quantization, Gemma 4 is highly capable of generating structured tool calls matching our schema without outputting garbage markdown text.
• Multimodal Capability: By supporting both audio and image inputs natively, Gemma 4 enables our Vision and Reader profiles to read screenshots and look up saved allergen data, making on-device safety checks fast.
• Large Context Window: The 32K context window is crucial because we feed user settings, memory history, layout hierarchies, and tool definitions into the prompt. Gemma 4 holds all of this context on-device without collapsing.

The Genie Team

Artificial Intelligence has transformed modern customer support systems by enabling faster and smarter interactions. AI-powered assistants can now answer questions, automate workflows, and assist customers efficiently.

However, despite these advancements, most AI support systems still suffer from one major limitation:

They forget everything.

Traditional AI agents treat every conversation as a completely new interaction. Customers repeatedly explain the same issues, re-enter their details, and lose conversational context every time they return.

This creates:

• repetitive conversations
• poor customer experience
• lack of personalization
• inefficient support systems

So we built Novasas Agent, an AI-powered customer support system capable of remembering previous customer interactions and using contextual memory to provide personalized responses.

But instead of simply claiming our AI had memory, we designed a real-time side-by-side comparison dashboard to visually demonstrate the difference between:

• AI without memory
• AI with memory

This became the core concept of our project.

The Problem With Traditional AI Systems

Most AI support systems today work statelessly.
The workflow typically looks like this:

• Customer asks a question
• AI responds
• Conversation ends
• AI forgets everything

When the same customer returns later, the AI behaves as if it has never interacted with them before.
For example:

First Interaction
Customer:
“I was charged twice for my subscription this month.”
AI:
“I’m sorry to hear that. Let me help you.”
Follow-Up Interaction
Customer:
“I’m following up on my refund request.”
Traditional AI:
“Can you explain your issue again?”

This makes customer support feel robotic and repetitive.

We wanted our AI system to behave more like a real support agent — one that remembers customers and understands previous conversations.

Our Solution: Novasas Agent

To visually demonstrate the impact of memory, we designed a dual-chat dashboard where both systems operate side-by-side in real time.

This allowed users to instantly compare how memory changes AI behavior.

Tech Stack Used

We built our project using:

• Python
• Groq API
• GitHub
• Hindsight API

For the language model, we used:

self.model = “llama-3.3-70b-versatile”

This model provided fast inference and clean conversational responses for our AI assistant.

Building the AI Support Agent

The backend of Novasas Agent was built entirely using Python.
The AI receives:

• customer name
• customer ID
• customer issue
  and generates support responses using Groq-powered LLMs.

Initially, our memory system was implemented using a Python dictionary:
Python
self.customer_history = {}

This allowed the AI to temporarily store customer interactions during runtime.

Retrieving Customer Memory

One of the most important components of our project was retrieving previous customer interactions.
We implemented a memory retrieval function:

def get_customer_history(self, customer_id: str) -> str:
“””Get previous interactions for a customer”””

if customer_id in self.customer_history:
    history = self.customer_history[customer_id]
    return f"nPrevious interactions with this customer:n{history}"

return "nThis is the customer's first interaction with us."

This allowed the AI agent to:

• identify returning customers
• retrieve earlier interactions
• provide contextual responses

Saving Customer Interactions

To maintain conversational memory, we created a function that stores customer interactions dynamically.

def save_interaction(self, customer_id: str,
customer_name: str,
message: str,
response: str):

interaction = f"- {customer_name}: {message}n Agent: {response}"

if customer_id not in self.customer_history:
    self.customer_history[customer_id] = ""

self.customer_history[customer_id] += interaction + "n"

Every interaction gets appended to the customer’s history, enabling the AI to remember previous conversations.

Building the Memory-Enabled AI

Our memory-enabled support function retrieves customer history before generating a response.

def handle_support_request(self,
customer_name: str,
customer_id: str,
message: str) -> str:

customer_context = self.get_customer_history(customer_id)

The retrieved context is injected directly into the AI prompt.
This enables the model to:

• recognize returning customers
• recall earlier issues
• generate more personalized support responses

Building the Stateless AI Version

To demonstrate the limitations of traditional AI systems, we also created a version without memory.

def handle_support_request_no_memory(
self,
customer_name: str,
customer_id: str,
message: str) -> str:

customer_context = "nThis is the customer's first interaction with us."

Unlike the memory-enabled version, this system always treats users as completely new customers.

This comparison became one of the strongest parts of our project.
Designing the Dual-Chat Dashboard
Instead of building a simple chatbot interface, we wanted users to instantly understand the importance of AI memory.

So we created a dual-chat dashboard:
Left Side — With Memory

• retrieves customer history
• remembers previous interactions
• provides personalized replies

Right Side — Without Memory

• generic responses
• no contextual understanding
• stateless behavior

This side-by-side visual demonstration clearly shows how memory improves customer experience.

First Interaction

During the first interaction, both AI systems respond similarly because no previous context exists yet.

The customer reports a billing issue:

“I was charged twice for my subscription this month.”

The memory-enabled system stores the interaction for future retrieval.

Second Interaction

When the customer returns later and asks:

“Following up on my refund for the double charges issue”

the difference becomes obvious.

With Memory

The AI remembers:

• previous billing issue
• transaction details
• earlier conversation context
  and responds accordingly.

Without Memory

The AI treats the customer as completely new and asks for information again.
This visual comparison became the highlight of our project.

Challenges We Faced

Like most engineering projects, we encountered several technical challenges during development.

1. Memory Reset After Restart
   Initially, customer memory was stored only in runtime variables.
   Whenever the application restarted, all memory disappeared.
   This meant the AI only had temporary conversational memory instead of true persistent memory.

2. API Handling
   Handling API responses consistently was another challenge.
   We had to carefully manage:

• API calls
• response formatting
• conversation flow
• contextual prompts
  to ensure smooth user interactions.

Why Persistent Memory Matters
Modern AI systems are becoming more conversational, but most still fail to maintain long-term context.
Persistent memory can significantly improve:

• customer support
• AI assistants
• educational chatbots
• productivity systems

By enabling AI systems to remember previous interactions, users receive:

• personalized responses
• smoother conversations
• reduced repetition
• more human-like interactions

Our project demonstrates how even simple memory systems can dramatically improve AI behavior.

Future Improvements

Although our current prototype works successfully, we plan to improve the project further with:

• Multi-user authentication
• Voice-based support
• Advanced memory databases
• Cloud deployment
• Long-term semantic memory
• Smarter retrieval systems

These upgrades would make the system more scalable and production-ready.

Final Thoughts

One of the biggest weaknesses of traditional AI systems is their inability to maintain memory and context across interactions.

With Novasas Agent, our goal was to demonstrate how persistent memory can improve customer support experiences in a simple yet impactful way.

By visually comparing AI systems with and without memory, we were able to clearly show why contextual AI matters.

This project taught us that AI should not just respond.

AI should remember.

Because memory is what makes interactions feel intelligent, personal, and human.

Your inbox is data. Query it like data.

The 2-Minute Version:

# Install and ingest your Gmail
brew tap surveilr/tap && brew install surveilr
surveilr admin init -d email.db
surveilr ingest imap 
  -u [email protected] 
  -p "your-app-password" 
  -a imap.gmail.com 
  -d email.db

# Query it
surveilr shell -d email.db

-- Find every email from a specific sender
SELECT subject, date, "from"
FROM emails
WHERE "from" LIKE '%@vendor.com%'
ORDER BY date DESC;

Your inbox is now a SQL database. Query it like any other data.

The Problem with Email Search

Gmail and Outlook have search boxes. They work for simple queries.

But try this:

• “Show me all emails from Q1 2024 where attachments were over 10MB”
• “Find email threads that mention both ‘invoice’ and ‘overdue'”
• “List all external recipients I’ve emailed in the last 6 months”
• “Which emails have attachments but no reply?”

Most email clients can’t answer these questions.

Here’s why:

Email search is designed for finding messages, not analyzing communication patterns.

surveilr turns your inbox into a SQLite database where these queries become trivial.

What surveilr Does

surveilr connects to any IMAP server and extracts everything into SQL tables:

• From, To, CC, BCC
• Subject lines and dates
• Message content and headers
• Attachments (with optional extraction)
• Thread relationships
• Folder structure

The output is standard SQLite. Use any SQL tool you want.

Step 1: Get an App Password

For Gmail:

1. Go to Google Account → App Passwords
2. Generate an app password
3. Copy it (you’ll use this instead of your regular password)

Step 2: Create a Database

surveilr admin init -d email.db

Standard SQLite file. Nothing fancy.

Step 3: Ingest Your Email

surveilr ingest imap 
  -d email.db 
  -u [email protected] 
  -p "your-app-password" 
  -a imap.gmail.com 
  -f "INBOX"

surveilr connects via IMAP and inserts messages into ur_ingest_session_imap_acct_folder_message.

Useful flags:

Filter by status:

--status unread    # Only unread messages
--status starred   # Starred/flagged
--status all       # Everything (default)

Extract attachments:

--extract-attachments uniform-resource  # Store in database

Show progress:

--progress  # Display download progress

Limit messages:

-b 1000  # Process up to 1000 messages

Step 4: Query Your Inbox

Open the SQL shell:

surveilr shell -d email.db

Now you can ask questions that Gmail search can’t answer.

Find all emails from a specific domain

SELECT
    "from",
    subject,
    date
FROM ur_ingest_session_imap_acct_folder_message
WHERE "from" LIKE '%@vendor.com%'
ORDER BY date DESC;

Note: from is quoted because it’s a SQL keyword.

Find emails with large attachments

SELECT
    m.subject,
    m."from",
    m.date,
    a.name AS attachment_name,
    a.size_bytes / 1024 / 1024 AS size_mb
FROM ur_ingest_session_imap_acct_folder_message m
JOIN ur_ingest_session_attachment a
  ON m.message_id = a.message_id
WHERE a.size_bytes > 10485760
ORDER BY a.size_bytes DESC;

Track email volume over time

SELECT
    DATE(date) AS day,
    COUNT(*) AS emails
FROM ur_ingest_session_imap_acct_folder_message
WHERE date > date('now', '-30 days')
GROUP BY DATE(date)
ORDER BY day;

Find emails you sent but never got a reply to

SELECT
    subject,
    "to",
    date
FROM ur_ingest_session_imap_acct_folder_message
WHERE "from" LIKE '%[email protected]%'
  AND message_id NOT IN (
    SELECT in_reply_to
    FROM ur_ingest_session_imap_acct_folder_message
    WHERE in_reply_to IS NOT NULL
  )
ORDER BY date DESC;

These queries are impossible in Gmail.

List Available Folders

Not sure what folders you have?

surveilr ingest imap 
  -u [email protected] 
  -p "your-app-password" 
  -a imap.gmail.com 
  --list-folders

Then ingest specific folders:

surveilr ingest imap 
  -u [email protected] 
  -p "your-app-password" 
  -a imap.gmail.com 
  -f "[Gmail]/Sent Mail" 
  -d email.db

Export Results

Need to share findings? Export as JSON:

surveilr shell -d email.db --cmd "
SELECT subject, "from", date
FROM ur_ingest_session_imap_acct_folder_message
WHERE subject LIKE '%invoice%'
" --output json > invoices.json

Or CSV for spreadsheet analysis:

surveilr shell -d email.db --cmd "
SELECT * FROM ur_ingest_session_imap_acct_folder_message
" --output csv > all-emails.csv

Why Not Just Export to CSV?

You could export your inbox to CSV or use a backup tool.

But:

Exports are point-in-time snapshots.
You can’t incrementally update them.

surveilr supports continuous ingestion.
Run it weekly and accumulate history over time.

CSVs don’t preserve relationships.
How do you link attachments to messages?

surveilr uses proper foreign keys.
Attachments reference their parent messages.

CSVs are hard to query.
You’re back to scripting with pandas or awk.

surveilr gives you SQL.
Join, filter, aggregate—all standard SQL.

Forensic Email Analysis

Once your inbox is queryable, you can investigate patterns:

Find emails that mention specific keywords across years

SELECT subject, "from", date
FROM emails
WHERE (subject LIKE '%confidential%' OR content LIKE '%confidential%')
  AND date > '2020-01-01'
ORDER BY date DESC;

Track who you email most frequently

SELECT
    "to",
    COUNT(*) AS email_count
FROM emails
WHERE "from" LIKE '%[email protected]%'
GROUP BY "to"
ORDER BY email_count DESC
LIMIT 20;

Find emails sent on weekends or late at night

SELECT subject, "to", date
FROM emails
WHERE strftime('%w', date) IN ('0', '6')  -- Sunday or Saturday
   OR CAST(strftime('%H', date) AS INTEGER) > 22
   OR CAST(strftime('%H', date) AS INTEGER) < 6
ORDER BY date DESC;

These insights are buried in your inbox. SQL makes them visible.

Real-World Uses

For Developers

• Track bug reports sent via email
• Find API key or credential leaks in sent mail
• Analyze support ticket volume
• Search across years of technical discussions

For Operations

• Audit external communications
• Track alert emails and response times
• Find recurring issues mentioned in email
• Monitor vendor correspondence

For Legal/E-Discovery

• Respond to discovery requests with SQL queries
• Prove communication timelines
• Find all emails related to a specific matter
• Extract conversations by date range

For Compliance (Oh, By the Way)

• HIPAA: Track communications containing PHI
• SOX: Maintain 7-year email records with queryable evidence
• GDPR: Respond to “right to access” requests with SQL
• E-Discovery: Legal hold and litigation support

But the real value is permanent, queryable communication history.

Open the Database in Other Tools

Because it’s SQLite, you can use any SQLite tool:

• DB Browser for SQLite — Visual inspection
• Datasette — Instant web UI
• Python pandas — pd.read_sql("SELECT * FROM emails", conn)
• DuckDB — Fast analytical queries
• Metabase — Build dashboards
• R / tidyverse — Data analysis workflows

You own the database. Query it however you want.

Security Best Practices

Use App Passwords, Not Your Main Password

Never put your main email password in scripts. Use app-specific passwords that you can revoke.

Store Passwords Securely

Use environment variables:

export IMAP_PASSWORD="your-app-password"
surveilr ingest imap -u [email protected] -p "$IMAP_PASSWORD" -a imap.gmail.com

Or use a password manager’s CLI:

surveilr ingest imap -u [email protected] -p "$(pass show gmail/app-password)" -a imap.gmail.com

Encrypt the Database

SQLite databases can be encrypted. Use tools like SQLCipher if you need encryption at rest.

Troubleshooting

“Authentication failed”?

• Make sure you’re using an app password, not your main password
• Verify IMAP is enabled in your email settings
• Check that 2FA isn’t blocking IMAP access

“Connection timed out”?

• Verify the IMAP server address
• Check if your firewall blocks port 993
• Some providers require enabling “less secure app access”

“No messages found”?

• Use --list-folders to see available folders
• Folder names are case-sensitive
• Gmail uses [Gmail]/All Mail not All Mail

What’s Next?

You just turned your inbox into a queryable database.

Now combine it with other data:

• Query Your File System — Join emails with file metadata
• Extract API Data — Add GitHub, Jira, or other sources

Or explore more:

• surveilr.com — Full documentation
• Installation Guide — Other platforms

The Bottom Line

Your inbox shouldn’t be a black box.

Email clients give you search bars and folders. That’s it.

surveilr gives you SQL.

Want to know:

• What you emailed last quarter?
• Who sends you the most messages?
• Which threads have attachments but no replies?
• How your communication patterns changed over time?

Just write a query.

Your email is now a SQLite database you own forever.

No export limits. No API restrictions. Just standard SQL.

Ready to query your inbox? Install surveilr and ingest your first folder.

Get Started →