Reference articles:

• Gemini API – Function Calling with Multimodal
• GitHub: linebot-gemini-multimodel-funcal
• Vertex AI – Multimodal Function Response
• Complete code GitHub

Background

I believe many people have used the combination of LINE Bot + Function Calling. When a user asks “What clothes did I buy last month?”, the Bot calls the database query function, retrieves the order data, and then Gemini answers based on that JSON:

Traditional process designed by developers:

User: "Help me see the jacket I bought before"
Bot: [Call get_order_history()]
Function returns: {"product_name": "Brown pilot jacket", "order_date": "2026-01-15", ...}
Gemini: "You bought a brown pilot jacket on January 15th for NT$1,890."

The answer is completely correct, but it always feels like something is missing—the user is talking about “that jacket,” and Gemini is just restating the text in the JSON, with no way to “confirm” what the jacket looks like. If there happen to be three jackets in the database, the AI can’t even determine which one is the one the user remembers.

AI can read text, but it can’t see pictures—this limitation has always been a blind spot in the traditional Function Calling architecture.

This problem was truly solved only after Gemini introduced Multimodal Function Response.

What is Multimodal Function Response?

The traditional Function Calling process is as follows:

[User message] → Gemini → [function_call] → [Execute function] → [Return JSON] → Gemini → [Text answer]

Multimodal Function Response changes that middle step. The function can not only return JSON, but also include images (JPEG/PNG/WebP) or documents (PDF) in the same response:

[User message] → Gemini → [function_call] → [Execute function] → [Return JSON + image bytes] → Gemini → [Text answer that has seen the image]

When Gemini generates the next round of answers, it can “see” both the structured data and the image returned by the function, thereby generating richer and more accurate responses.

The official currently supported media formats:

The application scenarios for this feature are very broad: e-commerce customer service (identifying product images), medical consultation (analyzing PDF inspection reports), design review (giving suggestions based on screenshots)… almost all scenarios that require “functions to return visual data for AI analysis” are applicable.

Project Goal

This time, I used Multimodal Function Response to create a LINE e-commerce customer service robot, demonstrating the following scenario:

User: “Help me see the jacket I bought before” Bot (traditional): “You bought a brown pilot jacket.” Bot (Multimodal): “From the photo, you can see that this is a brown pilot jacket, made of lightweight nylon, with metal zipper pockets on the sides. This is your January 15th order ORD-2026-0115, for a total of NT$1,890, and it has been delivered.” + Product photo

The difference is obvious: Gemini really “saw” the jacket, rather than just restating the text in the database.

Architecture Design

Why not use Google ADK?

Originally, this repo used Google ADK (Agent Development Kit) to manage the Agent. The Runner and Agent of ADK encapsulated the entire process of Function Calling, which was very convenient.

But Multimodal Function Response requires manually including image bytes in the parts of the function response, and ADK completely encapsulates this layer, so it can’t be intervened.

So this time, I directly used google.genai.Client to implement the iterative cycle of function calls myself:

# Old architecture (ADK)
runner = Runner(agent=root_agent, ...)
async for event in runner.run_async(...):
    ... # ADK handles all function calls for you, but you can't control the response content

# New architecture (directly use google.genai)
response = await client.aio.models.generate_content(
    model=model,
    contents=contents,
    config=types.GenerateContentConfig(tools=ECOMMERCE_TOOLS),
)
# Handle function calls yourself, include images yourself

Overall architecture

LINE User
    │
    ▼ POST /
FastAPI Webhook Handler
    │
    ▼
EcommerceAgent.process_message(text, line_user_id)
    │
    ├─ ① Call Gemini (with conversation history)
    │
    ├─ ② Gemini decides to call a tool → function_call
    │
    ├─ ③ _execute_tool()
    │ ├─ Execute query function (search_products / get_order_history / get_product_details)
    │ └─ Read real product photos in the img/ directory (Unsplash JPEG)
    │
    ├─ ④ Construct Multimodal Function Response
    │ └─ FunctionResponsePart(inline_data=FunctionResponseBlob(data=image_bytes))
    │
    ├─ ⑤ Call Gemini again (Gemini sees the image + data)
    │
    └─ ⑥ Return (ai_text, image_bytes)
    │
    ▼
LINE Reply:
  TextSendMessage(text=ai_text)
  ImageSendMessage(url=BOT_HOST_URL/images/{uuid}) ← FastAPI /images endpoint provides

How to get product images?

This demo uses real Unsplash clothing photography photos. Each of the five products corresponds to an actual photo of the item, stored in the img/ directory. The reading logic is very simple:

def generate_product_image(product: dict) -> bytes:
    """Read the product image and return JPEG bytes."""
    with open(product["image_path"], "rb") as f:
        return f.read()

Each product in PRODUCTS_DB has an image_path field pointing to the corresponding image file:

The image bytes read have two uses:

1. As FunctionResponseBlob to include for Gemini analysis—real photos allow Gemini to describe the actual fabric texture and tailoring details
2. Temporarily stored in the image_cache dict, provided to the LINE Bot for display through the FastAPI /images/{uuid} endpoint

Detailed explanation of the core code

Step 1: Define tools (FunctionDeclaration)

from google.genai import types

ECOMMERCE_TOOLS = [
    types.Tool(function_declarations=[
        types.FunctionDeclaration(
            name="get_order_history",
            description="Query the current user's order history",
            parameters=types.Schema(
                type=types.Type.OBJECT,
                properties={
                    "time_range": types.Schema(
                        type=types.Type.STRING,
                        description="Time range: all / last_month / last_3_months",
                        enum=["all", "last_month", "last_3_months"],
                    ),
                },
                required=[],
            ),
        ),
        # ... search_products, get_product_details
    ])
]

Step 2: Function call cycle (up to 5 iterations)

async def process_message(self, text: str, line_user_id: str):
    contents = self._get_history(line_user_id) + [
        types.Content(role="user", parts=[types.Part(text=text)])
    ]

    for _iteration in range(5): # Up to 5 times, to prevent infinite loops
        response = await self._client.aio.models.generate_content(
            model=self._model,
            contents=contents,
            config=types.GenerateContentConfig(
                system_instruction=_SYSTEM_INSTRUCTION,
                tools=ECOMMERCE_TOOLS,
            ),
        )

        model_content = response.candidates[0].content
        contents.append(model_content)

        # Find all function_call parts
        fc_parts = [p for p in model_content.parts if p.function_call and p.function_call.name]

        if not fc_parts:
            # No function call → final text response
            final_text = "".join(p.text for p in model_content.parts if p.text)
            break

        # Has function call → execute tool, include image
        tool_parts = []
        for fc_part in fc_parts:
            result_dict, image_bytes = _execute_tool(
                fc_part.function_call.name,
                dict(fc_part.function_call.args),
                line_user_id,
            )
            tool_parts.append(
                self._build_multimodal_response(fc_part.function_call.name, result_dict, image_bytes)
            )

        contents.append(types.Content(role="tool", parts=tool_parts))

Step 3: Construct Multimodal Function Response (the most critical step)

def _build_multimodal_response(self, func_name, result_dict, image_bytes):
    multimodal_parts = []

    if image_bytes:
        # ⚠️ Note: Use FunctionResponseBlob here, not types.Blob!
        multimodal_parts.append(
            types.FunctionResponsePart(
                inline_data=types.FunctionResponseBlob(
                    mime_type="image/jpeg",
                    data=image_bytes, # raw bytes, SDK handles base64 internally
                )
            )
        )

    return types.Part.from_function_response(
        name=func_name,
        response=result_dict, # Structured JSON data
        parts=multimodal_parts or None, # ← Image is here! Gemini can "see" it after receiving it
    )

Gemini will receive both result_dict (order JSON) and image_bytes (product image) in the next generate_content call, and the generated answer can therefore describe the visual content of the image.

Step 4: LINE Bot simultaneously returns text + image

# main.py

ai_text, image_bytes = await ecommerce_agent.process_message(msg_text, line_user_id)

reply_messages = [TextSendMessage(text=ai_text)]

if image_bytes:
    image_id = str(uuid.uuid4())
    image_cache[image_id] = image_bytes # Temporary storage
    image_url = f"{BOT_HOST_URL}/images/{image_id}" # FastAPI provides service
    reply_messages.append(
        ImageSendMessage(
            original_content_url=image_url,
            preview_image_url=image_url,
        )
    )

await get_line_bot_api().reply_message(event.reply_token, reply_messages)

LINE Bot’s reply_message supports returning multiple messages at once (up to 5), so text and images can be sent simultaneously.

Potholes

❌ Pitfall 1: FunctionResponseBlob is not Blob

The most common pitfall: When constructing multimodal image parts, you cannot use types.Blob, you must use types.FunctionResponseBlob:

# ❌ Error (will TypeError)
types.FunctionResponsePart(
    inline_data=types.Blob(mime_type="image/jpeg", data=image_bytes)
)

# ✅ Correct
types.FunctionResponsePart(
    inline_data=types.FunctionResponseBlob(mime_type="image/jpeg", data=image_bytes)
)

Although both have mime_type and data fields, the inline_data field type of FunctionResponsePart is FunctionResponseBlob, and Pydantic validation will directly reject Blob. You can confirm this with python -c "from google.genai import types; print(types.FunctionResponsePart.model_fields)".

❌ Pitfall 2: aiohttp.ClientSession cannot be created at the module level

The original code directly created aiohttp.ClientSession() at the module level:

# ❌ Old method: module level
session = aiohttp.ClientSession() # Will warn or error if there is no running event loop
async_http_client = AiohttpAsyncHttpClient(session)

When importing main.py in pytest tests, because there is no running event loop, RuntimeError: no running event loop will appear. The solution is to change to lazy initialization, and create it only when it is actually needed for the first time:

# ✅ New method: lazy init
_line_bot_api = None

def get_line_bot_api():
    global _line_bot_api
    if _line_bot_api is None:
        session = aiohttp.ClientSession() # Called within the async route handler, guaranteeing an event loop
        _line_bot_api = AsyncLineBotApi(channel_access_token, AiohttpAsyncHttpClient(session))
    return _line_bot_api

❌ Pitfall 3: LINE Bot needs HTTPS URL to send images

Gemini receives raw bytes, but LINE Bot’s ImageSendMessage requires a publicly accessible HTTPS URL.

The solution is to add a /images/{image_id} endpoint in FastAPI, temporarily store the read image bytes in the image_cache dict, and LINE gets the image through this endpoint:

@app.get("/images/{image_id}")
async def serve_image(image_id: str):
    image_bytes = image_cache.get(image_id)
    if image_bytes is None:
        raise HTTPException(status_code=404, detail="Image not found")
    return Response(content=image_bytes, media_type="image/jpeg")

Use ngrok to expose port 8000 for local development, and use the service URL directly after Cloud Run deployment.

Demo Display

Mock database (default data for demo)

The system has 5 built-in products (all with real Unsplash photos), and each LINE user automatically binds two demo orders when querying orders for the first time:

Scenario 1: “Help me see the jacket I bought before”

User sends: "Help me see the jacket I bought before"

[Gemini → function_call]
  get_order_history(time_range="all")

[_execute_tool execution]
  - get_order_history() returns two orders (P001, P003)
  - Read img/tobias-tullius-...-unsplash.jpg → Brown pilot jacket real photo bytes

[Multimodal Function Response]
  Part.from_function_response(
    name="get_order_history",
    response={"orders": [...], "order_count": 2},
    parts=[FunctionResponsePart(inline_data=FunctionResponseBlob(data=<photo>))]
  )

[Gemini responds after seeing the real photo]
  "From the photo, you can see that this is a brown pilot jacket, made of lightweight nylon with
   a glossy feel, and a metal zipper pocket on the left sleeve. This is your January 15, 2026
   order ORD-2026-0115, for a total of NT$1,890, status: delivered."

LINE displays: [Text] + [Brown pilot jacket real photo]

Scenario 2: “Are there any dark blue jackets?”

[Gemini → function_call]
  search_products(description="dark blue jacket", color="dark blue")

[Gemini sees the real photo of the P003 dark blue denim jacket]
  "Yes! This dark blue denim jacket (P003) in the photo features a retro stitching design,
   a lapel with metal buttons, and a very complete garment feel, priced at NT$1,490, with 8 in stock."

Scenario 3: “What are the features of the P004 knitted shawl?”

[Gemini → function_call]
  get_product_details(product_id="P004")

[Gemini sees the real photo of the beige knitted shawl]
  "The photo shows a beige handmade crochet shawl, with a V-neck design and tassels at the bottom,
   you can see the light lace-like mesh weave, elegant texture, priced at NT$1,290."

Traditional Function Response vs Multimodal Function Response

Analysis and Outlook

This implementation gave me a new understanding of Gemini’s Function Calling capabilities.

The problem that Multimodal Function Response truly solves is to allow AI agents to bring in visual information in the action of “calling an external system” itself, instead of first checking text and then uploading images separately. This will be an important basic capability in areas highly related to visuals, such as e-commerce, medicine, and design.

However, there are still a few limitations worth noting:

1. Image URLs cannot be used directly: Gemini’s FunctionResponseBlob requires raw bytes, and URLs cannot be filled in directly (this is different from bringing images directly in the prompt). If the image is originally a URL, you need to download it with requests.get() to bytes before passing it in.

2. No display_name can also be used: The official documentation examples have display_name and $ref JSON reference, but in actual testing in google-genai 1.49.0, it can also work normally without filling in display_name, and Gemini can still see and analyze the image.

3. Model limitations: The official mark supports the Gemini 3 series, but gemini-2.0-flash can also handle it normally in actual testing, and the API structure is the same.

There are many directions that can be extended in the future: let users send their own product photos for the Bot to compare, include PDF catalogs in the function response for Gemini to read directly, or let the Bot analyze the report images converted from DICOM in medical scenarios… As long as visual data can be obtained from external systems, Multimodal Function Response can make the AI’s answers more in-depth.

Summary

The focus of this LINE Bot implementation is only one sentence: Let the function response carry the image, and Gemini’s answer will be upgraded from “restating data” to “telling a story based on the picture”.

The core API is just these few lines, but it takes a lot of details to get the whole process working:

# The complete way for Gemini to see the image returned by the function
types.Part.from_function_response(
    name="get_order_history",
    response={"orders": [...]},
    parts=[
        types.FunctionResponsePart(
            inline_data=types.FunctionResponseBlob( # ← Not types.Blob!
                mime_type="image/jpeg",
                data=image_bytes,
            )
        )
    ],
)

The complete code is on GitHub, feel free to clone and play with it.

See you next time!

Reference articles:

• Gemini API tooling updates: context circulation, tool combos and Maps grounding for Gemini 3
• Google Places API (New) – searchNearby
• GitHub: linebot-spot-finder
• Complete code GitHub (Meeting Helper LINE Bot Spot Finder)

Background

The combination of LINE Bot + Gemini is already very common. Whether it’s using Google Search Grounding to let the model look up real-time information or using Function Calling to let the model call custom logic, they are both mature when used alone.

But what if you want to achieve both “map location context” and “query real ratings” in the same question?

Taking restaurant search as an example, the traditional approach usually looks like this:

User: "Help me find a hot pot restaurant nearby with a rating of 4 stars or above"

Solution A (using only Maps Grounding):
Gemini has map context, but the rating information is described by AI itself, and accuracy is not guaranteed.

Solution B (using only Places API):
You can get real ratings, but there is no map context, and Gemini doesn't know where the user is.

To have both, you usually need to make two API calls, or manually connect them yourself.

AI can search maps and call external APIs, but doing both in a single call—has always been an awkward blank in the old Gemini API architecture.

Until March 17, 2026, Google released Gemini API Tooling Updates (by Mariano Cocirio), which provided an official solution to this problem.

What are Tool Combinations?

Google announced three core features in this update:

1. Tool Combinations Developers can now attach built-in tools (such as Google Search, Google Maps) and custom Function Declarations simultaneously in a single Gemini API call. The model decides which tool to call and when to call it, and finally integrates the results to generate an answer.

2. Maps Grounding Gemini can now directly perceive map data, not just text descriptions of “location”, but truly has spatial context—knowing where the user is and what’s nearby.

3. Context Circulation Allows the context between multi-turn tool calls to flow naturally, and the model can fully remember the results of the first tool call when making the second call.

The key to this change is:

# Old approach (two tools cannot coexist)
types.Tool(google_search=types.GoogleSearch())
types.Tool(function_declarations=[MY_FN])

# New approach (the same Tool object, both coexist)
types.Tool(
    google_maps=types.GoogleMaps(),
    function_declarations=[MY_FN],
)

One line of modification opens up a whole new combination method.

Project Goal

This time, I used Tool Combinations to transform the existing linebot-spot-finder, upgrading it from “only Maps Grounding for rough answers” to “Google Maps context + Places API real data”:

After the user sends their GPS location, they enter: “Please find a hot pot restaurant with a rating of 4 stars or above, suitable for group dining, and list the name, address, and review summary.”

Bot (old Maps Grounding): “There are several hot pot restaurants nearby, and the ratings are good.” (AI describes it itself, which may not be accurate)

Bot (new Tool Combo): “Lao Wang Hot Pot | 100 Shimin Avenue, Xinyi District, Taipei City | Rating 4.6 (312) | Reviews: Large portions, great value for money, suitable for group dining; efficient service, fast serving.”

The difference is: Gemini now receives both map context (where you are) and the real structured data (rating numbers, review text) from the Places API, so the answer changes from a “vague description” to “informed information”.

Architecture Design

Overall Message Flow

LINE User sends GPS location
    │
    ▼
handle_location() → session.metadata stores lat/lng
    │
    └──► Returns Quick Reply (restaurant / gas station / parking lot)

LINE User sends text question (e.g. "Find a hot pot restaurant with a rating of 4 stars or above")
    │
    ▼
handle_text()
    │
    ├── session has lat/lng?
    │ Yes → tool_combo_search(query, lat, lng) ← Focus of this article
    │ No → fallback: Gemini Chat + Google Search
    │
    └──► Returns natural language answer

Tool Combo Agentic Loop

tool_combo_search(query, lat, lng)
         │
         ▼
  Step 1: generate_content()
  tools = [google_maps + search_nearby_restaurants]
         │
         ▼
  response.candidates[0].content.parts has function_call?
       ╱ ╲
      Yes   No
      │     │
      ▼     ▼
  _execute_function()  Directly returns response.text
  → _call_places_api()
    (Places API searchNearby)
    Returns rating, address, reviews
      │
      ▼
  Collect into a single Content(role="user")
  Add to history
      │
      ▼
  Step 3: generate_content(contents=history)
  Gemini integrates map context + Places data
      │
      ▼
  Returns final.text

Why not put lat/lng in Function Declaration?

This is an important design decision.

If you add lat/lng to the parameters of SEARCH_NEARBY_RESTAURANTS_FN, Gemini will fill in the coordinates itself—but it fills in the “approximate location” inferred from the conversation, not the user’s actual GPS coordinates, and the error can be as high as several kilometers.

The correct approach is to let the Python dispatcher extract the precise coordinates from session.metadata and inject them:

def _execute_function(name: str, args: dict, lat: float, lng: float):
    if name == "search_nearby_restaurants":
        return _call_places_api(
            lat=lat, lng=lng, # ← Inject from session, don't let Gemini guess
            keyword=args.get("keyword", ""),
            min_rating=float(args.get("min_rating", 4.0)),
        )

Core Code Details

Step 1: Define Function Declaration

from google.genai import types

SEARCH_NEARBY_RESTAURANTS_FN = types.FunctionDeclaration(
    name="search_nearby_restaurants",
    description=(
        "Search for nearby restaurants using Google Places API, and return the rating, address, and user reviews."
        "lat/lng is automatically included by the system and does not need to be provided."
    ),
    parameters=types.Schema(
        type=types.Type.OBJECT,
        properties={
            "keyword": types.Schema(
                type=types.Type.STRING,
                description="Restaurant type or keyword, such as: hot pot, hot pot, Italian",
            ),
            "min_rating": types.Schema(
                type=types.Type.NUMBER,
                description="Minimum rating threshold (1–5), default 4.0",
            ),
            "radius_m": types.Schema(
                type=types.Type.INTEGER,
                description="Search radius (meters), default 1000",
            ),
        },
    ),
)

The description clearly tells the model “lat/lng is included by the system”, avoiding the model filling in the coordinates itself in the args.

Step 2: Places API Call

import httpx

PLACES_API_URL = "https://places.googleapis.com/v1/places:searchNearby"
PLACES_FIELD_MASK = (
    "places.displayName,"
    "places.rating,"
    "places.userRatingCount,"
    "places.formattedAddress,"
    "places.reviews"
)

def _call_places_api(lat, lng, keyword="", min_rating=4.0, radius_m=1000):
    body = {
        "includedTypes": ["restaurant"],
        "maxResultCount": 5,
        "locationRestriction": {
            "circle": {
                "center": {"latitude": lat, "longitude": lng},
                "radiusMeters": radius_m,
            }
        },
    }

    response = httpx.post(
        PLACES_API_URL,
        headers={
            "X-Goog-Api-Key": os.getenv("GOOGLE_MAPS_API_KEY"),
            "X-Goog-FieldMask": PLACES_FIELD_MASK,
        },
        json=body,
        timeout=10.0,
    )
    response.raise_for_status()
    data = response.json()

    restaurants = []
    for place in data.get("places", []):
        rating = place.get("rating", 0)
        if rating < min_rating:
            continue
        reviews = [
            r["text"]["text"]
            for r in place.get("reviews", [])[:3]
            if r.get("text", {}).get("text")
        ]
        restaurants.append({
            "name": place["displayName"]["text"],
            "address": place.get("formattedAddress", ""),
            "rating": rating,
            "rating_count": place.get("userRatingCount", 0),
            "reviews": reviews,
        })

    return {"restaurants": restaurants}

Step 3: Tool Combo Main Function (Agentic Loop)

async def tool_combo_search(query: str, lat: float, lng: float) -> str:
    client = genai.Client(
        vertexai=True,
        project=os.getenv("GOOGLE_CLOUD_PROJECT"),
        location=os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1"),
        http_options=types.HttpOptions(api_version="v1"),
    )

    enriched_query = (
        f"User's current location: latitude {lat}, longitude {lng}.n"
        f"Please answer in traditional Chinese using Taiwanese terminology, and do not use markdown format.nn"
        f"Question: {query}"
    )

    tool_config = types.GenerateContentConfig(
        tools=[
            types.Tool(
                google_maps=types.GoogleMaps(), # ← Maps grounding
                function_declarations=[SEARCH_NEARBY_RESTAURANTS_FN], # ← Places API
            )
        ],
    )

    # ── Step 1 ──────────────────────────────────────────────────────
    response = client.models.generate_content(
        model=TOOL_COMBO_MODEL,
        contents=enriched_query,
        config=tool_config,
    )

    if not response.candidates:
        return response.text or "（Unable to get a reply）"

    history = [
        types.Content(role="user", parts=[types.Part(text=enriched_query)]),
        response.candidates[0].content,
    ]

    # ── Step 2：Processing function_call ──────────────────────────────────
    function_response_parts = []
    for part in response.candidates[0].content.parts:
        if part.function_call:
            fn = part.function_call
            result = _execute_function(fn.name, dict(fn.args or {}), lat, lng)
            function_response_parts.append(
                types.Part(
                    function_response=types.FunctionResponse(
                        id=fn.id, name=fn.name, response=result,
                    )
                )
            )

    if function_response_parts:
        history.append(types.Content(role="user", parts=function_response_parts))

        # ── Step 3 ────────────────────────────────────────────────────
        final = client.models.generate_content(
            model=TOOL_COMBO_MODEL,
            contents=history,
            config=tool_config,
        )
        return final.text or "（Unable to get a reply）"

    return response.text or "（Unable to get a reply）"

Pitfalls Encountered

❌ Pitfall 1: Part.from_function_response() does not accept the id parameter

This is the easiest pitfall to step into this time, and the error only explodes when real model calls are made, and unit tests almost never detect it.

Originally, I wrote it like this, referring to the official example:

# ❌ Error——TypeError occurs at runtime
types.Part.from_function_response(
    id=fn.id, # ← This parameter does not exist!
    name=fn_name,
    response=result,
)

The actual signature of from_function_response is:

(*, name: str, response: dict, parts: Optional[list] = None) -> Part

There is no id parameter at all. Every time the model actually triggers a function_call, the program will throw a TypeError at this line, and then silently enter the except of Step 3, returning an error message, and the results of the Places API are never truly returned to Gemini.

The correct way is to directly construct types.FunctionResponse:

# ✅ Correct
types.Part(
    function_response=types.FunctionResponse(
        id=fn.id,
        name=fn_name,
        response=result,
    )
)

You can immediately confirm the parameter list with python -c "from google.genai import types; help(types.Part.from_function_response)".

❌ Pitfall 2: include_server_side_tool_invocations=True causes Pydantic to explode

I thought I should add this parameter after seeing the official documentation example:

# ❌ Error
types.GenerateContentConfig(
    tools=[...],
    include_server_side_tool_invocations=True, # ← The installed SDK version does not support it
)

In google-genai 1.49.0, this field is not in the model fields of GenerateContentConfig, and Pydantic will directly throw an extra_forbidden validation error. Just remove it, and the function is completely normal.

❌ Pitfall 3: textQuery is a parameter of searchText, not searchNearby

I thought “if there is a keyword, then bring it into the Places API”, and intuitively added it to the request body:

# ❌ Error——Invalid field for searchNearby endpoint
if keyword:
    body["textQuery"] = keyword

searchNearby only accepts fields such as includedTypes, locationRestriction; textQuery is a parameter of the searchText endpoint. Adding this field will not report an error (in some versions), but the keyword will not take effect at all.

The correct approach is to leave the keyword in the description of the Function Declaration for Gemini to refer to, let the model translate the intent to enriched_query, let Maps Grounding handle the keyword semantics, and Places API is only responsible for returning real rating data.

❌ Pitfall 4: No guard for response.candidates[0]

When the model encounters security filtering, RECITATION, or other abnormal termination, candidates may be an empty list, and then directly response.candidates[0] is IndexError.

# ❌ No guard
history = [
    types.Content(role="user", parts=[types.Part(text=enriched_query)]),
    response.candidates[0].content, # ← If candidates is empty, it will explode
]

# ✅ Add guard
if not response.candidates:
    return response.text or "（Unable to get a reply）"

history = [...]

Demo Display

Scenario 1: “Find a hot pot restaurant with a rating of 4 stars or above for group dining”

User sends: GPS location (Xinyi District, Taipei City, 25.0441, 121.5598)

User enters: "Please find a hot pot restaurant with a rating of 4 stars or above, suitable for group dining, and list the name, address, and review summary."

[Step 1: Gemini receives query + map context]
  → Detects the need for restaurant data, emit function_call:
    search_nearby_restaurants(keyword="hot pot", min_rating=4.0)

[Step 2: Python calls Places API]
  → lat=25.0441, lng=121.5598 injected from session
  → Returns 3 restaurants with a rating ≥ 4.0, including review text

[Step 3: Gemini integrates Maps context + Places data]
  → "Lao Wang Hot Pot｜100 Shimin Avenue, Xinyi District｜⭐ 4.6 (312)
      Review summary: Large portions, great value for money, a top choice for friends to dine; fast service, fresh dishes.
     ... (3 restaurants in total)"

Scenario 2: “Are there any high-value Japanese restaurants?”

User enters: "Are there any high-value Japanese restaurants nearby?"

[Step 1: Gemini]
  → function_call: search_nearby_restaurants(keyword="Japanese cuisine", min_rating=4.0)

[Step 2: Places API]
  → Returns 2 Japanese restaurants that meet the rating criteria

[Step 3: Gemini]
  → "There are two recommendations:
      Washoku ○○｜...｜⭐ 4.4｜Reviews: Weekday lunch set is only 280 yuan, very fresh.
      ..."

Demo Script Quick Test

No need for LINE Bot, directly on the local machine:

# Only test Tool Combo (main function)
python demo.py combo

# Run all three functions
python demo.py all

Old Architecture vs. New Architecture

Analysis and Outlook

This implementation has given me a clearer understanding of the potential of Gemini Tool Combinations.

The problem that Tool Combinations truly solves is that Grounding and Function Calling are no longer mutually exclusive. Previously, to achieve “map context + real external data”, you could only manually connect two APIs yourself at the application layer, or use Gemini’s text generation to “simulate” external data (unreliable). Now the model itself knows when to use map context and when to call the Places API, and developers only need to attach the tools.

However, there are also a few things to note about this implementation:

1. lat/lng injection mode is very important: You can’t let the model guess the coordinates itself, you must inject them from the session, otherwise the positioning accuracy will be very poor. This mode also applies to all function calling scenarios that “have session status”.

2. The cost of two generate_content calls: The agentic loop of Tool Combo requires two model calls, and the token consumption is about 1.5–2 times that of a single call. This needs to be especially considered for scenarios with high latency requirements.

3. SDK version differences: Different versions of google-genai have different support for the fields of GenerateContentConfig, and new fields like include_server_side_tool_invocations should be used after confirming the version number, otherwise Pydantic validation errors are hard to track.

Future directions that can be extended:

• Connect the Postback quick replies (click the “Find Restaurant” button) to Tool Combo, so that each entry can get real ratings
• Add the searchText endpoint to support more complex keyword searches (e.g. Michelin recommendations)
• Tool Combo combined with other built-in tools (such as google_search) to achieve more complex multi-tool chaining

Summary

The core concept of this modification is only one sentence: Put Google Maps grounding and the Places API function tool in the same types.Tool, and Gemini will coordinate the two in a single conversation.

The key code is only these few lines:

# This is all the magic of Tool Combo
types.Tool(
    google_maps=types.GoogleMaps(), # ← Maps context
    function_declarations=[SEARCH_NEARBY_RESTAURANTS_FN], # ← Places API
)

But to make it really work, you also need to pay attention to: the construction method of FunctionResponse, the guard of candidates, the correct fields of the Places API endpoint, and the injection of lat/lng from the session instead of letting the model guess.

The complete code is on GitHub, feel free to clone and play with it.

See you next time!

Background

Google released Gemini 3.1 Flash Live at the end of March 2026 March, focusing on “making audio AI more natural and reliable.” This model is specifically designed for real-time two-way voice conversations, with low latency, interruptibility, and multi-language support.

I happened to have a LINE Bot project (linebot-helper-python) on hand, which already handles text, images, URLs, PDFs, and YouTube, but completely ignores voice messages:

User sends a voice message
Bot: (Silence)

This time, I’ll add voice support and share a few pitfalls I encountered.

Design Decision: Flash Live or Standard Gemini API?

The first question: Gemini 3.1 Flash Live is designed for real-time streaming, but LINE’s voice messages are pre-recorded m4a files, not real-time audio streams.

Using Flash Live to process pre-recorded files is like using a live streaming camera to take photos – technically feasible, but the wrong tool.

Decided to use the standard Gemini API – directly passing the audio bytes as inline data, and getting the transcribed text in one call. It’s simpler and more suitable for this scenario.

Architecture Design

Integration Approach

This repo already has a complete Orchestrator architecture, which automatically routes to different Agents (Chat, Content, Location, Vision, GitHub) based on the message content. The goal for voice messages is clear:

Convert voice to text, and then treat it as a regular text message and pass it into the Orchestrator – allowing all existing features to automatically support voice input.

User says “Help me search for nearby gas stations” → transcribed into text → Orchestrator determines it’s a location query → LocationAgent processes it. No need to implement separate logic for voice.

Complete Flow

User sends AudioMessage (m4a)
    │
    ▼ handle_audio_message()
    │
    ├─ ① LINE SDK downloads audio bytes
    │ get_message_content(message_id) → iter_content()
    │
    ├─ ② Gemini transcription
    │ tools/audio_tool.py → transcribe_audio()
    │ model: gemini-3.1-flash-lite-preview
    │
    ├─ ③ Reply #1: "You said: {transcription}"
    │ reply_message() (consumes reply token)
    │
    └─ ④ Reply #2: Orchestrator routing
            handle_text_message_via_orchestrator(push_user_id=user_id)
            ↓
            push_message() (reply token already used, use push instead)

Why two replies?

The replies are divided into two parts to let the user see the transcription result immediately, without waiting for the Orchestrator to finish processing to know if the Bot understood what they said.

Core Code Explanation

Step 1: Audio Transcription Tool (tools/audio_tool.py)

from google import genai
from google.genai import types

TRANSCRIPTION_MODEL = "gemini-3.1-flash-lite-preview"

async def transcribe_audio(audio_bytes: bytes, mime_type: str = "audio/mp4") -> str:
    """
    Transcribe audio bytes to text using Gemini.
    LINE voice messages are always m4a, MIME type is always audio/mp4.
    """
    client = genai.Client(
        vertexai=True,
        project=os.getenv("GOOGLE_CLOUD_PROJECT"),
        location=os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1"),
    )

    audio_part = types.Part.from_bytes(data=audio_bytes, mime_type=mime_type)

    response = await client.aio.models.generate_content(
        model=TRANSCRIPTION_MODEL,
        contents=[
            types.Content(
                role="user",
                parts=[
                    audio_part,
                    types.Part(text="Please transcribe the above audio content into text completely, preserving the original language, and do not add any explanations or prefixes."),
                ],
            )
        ],
    )

    return response.text or ""

Design principle: The function itself does not catch exceptions, allowing the upper-level handler to handle error responses uniformly.

Step 2: Handler Main Flow (main.py)

async def handle_audio_message(event: MessageEvent):
    """Handle audio (voice) messages — transcribe and route through Orchestrator."""
    user_id = event.source.user_id
    replied = False # Track if the reply token has been used
    try:
        # Download audio
        message_content = await line_bot_api.get_message_content(event.message.id)
        audio_bytes = b""
        async for chunk in message_content.iter_content():
            audio_bytes += chunk

        # Transcription
        transcription = await transcribe_audio(audio_bytes)

        # Empty transcription (silent or too short)
        if not transcription.strip():
            await line_bot_api.reply_message(
                event.reply_token,
                [TextSendMessage(text="Unable to recognize voice content, please re-record.")]
            )
            return

        # Reply #1: Let the user confirm the transcription result (consumes reply token)
        await line_bot_api.reply_message(
            event.reply_token,
            [TextSendMessage(text=f"You said: {transcription.strip()}")]
        )
        replied = True

        # Reply #2: Send to Orchestrator, using push_message (token already used)
        await handle_text_message_via_orchestrator(
            event, user_id,
            text=transcription.strip(),
            push_user_id=user_id,
        )

    except Exception as e:
        logger.error(f"Error handling audio for {user_id}: {e}", exc_info=True)
        error_text = LineService.format_error_message(e, "processing voice message")
        error_msg = TextSendMessage(text=error_text)
        if replied:
            # reply token has been consumed, use push instead
            await line_bot_api.push_message(user_id, [error_msg])
        else:
            await line_bot_api.reply_message(event.reply_token, [error_msg])

Step 3: Enabling Orchestrator to Support External Text Input

The original handle_text_message_via_orchestrator directly reads event.message.text. AudioMessage doesn’t have .text, so add two optional parameters:

async def handle_text_message_via_orchestrator(
    event: MessageEvent,
    user_id: str,
    text: str = None, # ← External text input (voice transcription)
    push_user_id: str = None, # ← Use push_message when set
):
    msg = text if text is not None else event.message.text.strip()
    try:
        result = await orchestrator.process_text(user_id=user_id, message=msg)
        response_text = format_orchestrator_response(result)
        reply_msg = TextSendMessage(text=response_text)

        if push_user_id:
            await line_bot_api.push_message(push_user_id, [reply_msg])
        else:
            await line_bot_api.reply_message(event.reply_token, [reply_msg])
    except Exception as e:
        error_msg = TextSendMessage(text=LineService.format_error_message(e, "processing your question"))
        if push_user_id:
            await line_bot_api.push_message(push_user_id, [error_msg])
        else:
            await line_bot_api.reply_message(event.reply_token, [error_msg])

text is not None (instead of text or ...) is intentional – in case the voice transcription results in an empty string, allow the empty string to pass through (and then be intercepted by the upper-level if not transcription.strip()), instead of falling back to event.message.text.

Pitfalls Encountered

❌ Pitfall 1: Part.from_text() does not accept positional arguments

The first TypeError encountered:

# ❌ Error (TypeError: Part.from_text() takes 1 positional argument but 2 were given)
types.Part.from_text(
    "Please transcribe the above audio content into text completely, preserving the original language, and do not add any explanations or prefixes."
)

# ✅ Correct
types.Part(text="Please transcribe the above audio content into text completely, preserving the original language, and do not add any explanations or prefixes.")

In this version of the SDK, text in Part.from_text() is a keyword argument, or use the Part(text=...) constructor directly for more safety.

❌ Pitfall 2: LINE reply token can only be used once

LINE’s reply token is one-time use. Once reply_message() is called, the token is invalidated.

This project’s voice flow will call twice:

1. Reply #1 (display transcription text) → consumes token
2. Reply #2 (Orchestrator result) → token is invalid, will receive LINE 400 error

The solution is to have the Orchestrator handler support push_message mode (via the push_user_id parameter), and Reply #2 changes to push_message.

Error handling should also be noted: if Orchestrator throws an exception after Reply #1 succeeds, the reply_message cannot be used in the except block, and it also needs to be changed to push_message. This is the purpose of the replied flag in the code.

❌ Pitfall 3: Gemini Flash Live is not suitable for pre-recorded files

Not a real “pitfall”, but worth clarifying:

Gemini 3.1 Flash Live is designed for real-time two-way streaming, which has the overhead of connection establishment and streaming protocols. LINE voice messages are complete pre-recorded m4a files, which can be processed once.

Using client.aio.models.generate_content() directly to pass inline audio bytes is simpler, and the delay is not bad. Leave Flash Live for scenarios that truly require real-time conversations.

Effect Demonstration

Scenario 1: Voice Command Query

User sends: [Voice] Help me search for cafes near Taipei Main Station

Bot Reply #1: You said: Help me search for cafes near Taipei Main Station
Bot Reply #2: [LocationAgent replies with a list of nearby cafes]

Scenario 2: Voice Question

User sends: [Voice] What's the difference between Gemini and GPT-4

Bot Reply #1: You said: What's the difference between Gemini and GPT-4
Bot Reply #2: [ChatAgent with Google Search Grounding replies with comparison results]

Scenario 3: Voice Send URL

User sends: [Voice] Help me summarize this article https://example.com/article

Bot Reply #1: You said: Help me summarize this article https://example.com/article
Bot Reply #2: [ContentAgent fetches and summarizes the article]

The text transcribed from voice goes directly into the Orchestrator, and all existing URL detection and intent determination work as usual, with zero extra logic.

Traditional Text Input vs. Voice Input

Analysis and Outlook

What I am most satisfied with in this integration is that I hardly need to change the Orchestrator itself. As long as the voice is converted to text at the input end, all the routing logic, Agent calls, and error handling are automatically inherited.

Gemini’s multimodal audio understanding performs very stably in this scenario – Traditional Chinese, Taiwanese accents, and sentences mixed with English can basically be transcribed accurately.

Future directions for extension:

• Multi-language automatic detection: Tell Gemini to preserve the original language during transcription, Japanese voice → Japanese transcription, and then the Orchestrator decides whether to translate
• Group voice support: Currently limited to 1:1, voice messages in groups are temporarily ignored
• Long recording summary: Recordings exceeding a certain length go directly to ContentAgent for summarization, instead of being treated as commands

Extension: 🔊 Read Summary Aloud – Make the Bot Speak

Voice recognition allows the Bot to “understand” what the user is saying. After this is done, the next question naturally arises:

Can the Bot respond by speaking?

The Gemini Live API has a setting response_modalities: ["AUDIO"], which can directly output an audio PCM stream. I connected it to another scenario – reading summaries aloud.

Function Design

Each time the Bot summarizes a URL, YouTube, or PDF, a “🔊 Read Aloud” QuickReply button will appear below the message. When the user presses it, the Bot sends the summary text into Gemini Live TTS, converts the PCM audio to m4a, and then sends it back using AudioSendMessage.

URL summary complete
    │
    ▼ [🔊 Read Aloud] QuickReply button
    │
User presses the button → PostbackEvent
    │
    ▼ handle_read_aloud_postback()
    │
    ├─ ① Retrieve the summary text from summary_store (10 minutes TTL)
    │
    ├─ ② Gemini Live API → PCM audio
    │ model: gemini-live-2.5-flash-native-audio
    │ response_modalities: ["AUDIO"]
    │
    ├─ ③ ffmpeg transcoding: PCM → m4a
    │ s16le, 16kHz, mono → AAC
    │
    └─ ④ AudioSendMessage sent to the user
            original_content_url: /audio/{uuid}
            duration: {ms}

Core Code (tools/tts_tool.py)

LIVE_MODEL = "gemini-live-2.5-flash-native-audio"

async def text_to_speech(text: str) -> tuple[bytes, int]:
    client = genai.Client(vertexai=True, project=VERTEX_PROJECT, location="us-central1")
    config = {"response_modalities": ["AUDIO"]}

    async with client.aio.live.connect(model=LIVE_MODEL, config=config) as session:
        await session.send_client_content(
            turns=types.Content(role="user", parts=[types.Part(text=text)]),
            turn_complete=True,
        )
        pcm_chunks = []
        async for message in session.receive():
            if message.server_content and message.server_content.model_turn:
                for part in message.server_content.model_turn.parts:
                    if part.inline_data and part.inline_data.data:
                        pcm_chunks.append(part.inline_data.data)
            if message.server_content and message.server_content.turn_complete:
                break

    pcm_bytes = b"".join(pcm_chunks)
    duration_ms = int(len(pcm_bytes) / 32000 * 1000) # 16kHz × 16-bit mono

    # PCM → m4a (temp file mode, avoid moov atom problem)
    with tempfile.NamedTemporaryFile(suffix=".pcm", delete=False) as f:
        f.write(pcm_bytes)
        pcm_path = f.name
    m4a_path = pcm_path.replace(".pcm", ".m4a")
    subprocess.run(
        ["ffmpeg", "-y", "-f", "s16le", "-ar", "16000", "-ac", "1",
         "-i", pcm_path, "-c:a", "aac", m4a_path],
        check=True, capture_output=True,
    )
    with open(m4a_path, "rb") as f:
        return f.read(), duration_ms

Pitfalls of Read Aloud Function

❌ Pitfall 4: Completely Different Model Name

The first attempt at Gemini Live TTS was:

LIVE_MODEL = "gemini-3.1-flash-live-preview"

Following the inference of gemini-3.1-flash-lite-preview used for voice recognition, the result was a direct 1008 policy violation:

Publisher Model `projects/line-vertex/locations/global/publishers/google/
models/gemini-3.1-flash-live-preview` was not found

Listing the available models on Vertex AI revealed that the model naming rules for Live/native audio are completely different:

# ✅ Correct
LIVE_MODEL = "gemini-live-2.5-flash-native-audio"

There is no Live version of Gemini 3.1 on Vertex AI. The Live/native audio feature is currently the 2.5 generation, and the naming format is gemini-live-{version}-{variant}-native-audio, which is completely separate from the general model gemini-{version}-flash-{variant}.

❌ Pitfall 5: GOOGLE_CLOUD_LOCATION=global Causes Live API to Disconnect

After changing to the correct model name, the error message was still the same:

Publisher Model `projects/line-vertex/locations/global/...` was not found

This time the model name was correct, but locations/global was strange – we clearly set us-central1.

Investigating the source code of the Google GenAI SDK revealed:

# _api_client.py
self.location = location or env_location
if not self.location and not self.api_key:
    self.location = 'global' # ← here

location or env_location – if the passed-in location is an empty string, it will fall back to global.

The root cause of the problem is the environment variable of Cloud Run:

{ "name": "GOOGLE_CLOUD_LOCATION", "value": "global" }

GOOGLE_CLOUD_LOCATION was set to the "global" string. os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1") did not get "us-central1", but "global" – then the SDK obediently connected to the global endpoint, but gemini-live-2.5-flash-native-audio does not have BidiGenerateContent support in global.

Solution: Hardcode the location of the Live API, and don’t read from the env var:

# ❌ Affected by GOOGLE_CLOUD_LOCATION=global
VERTEX_LOCATION = os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1")

# ✅ Hardcoded, not affected by env var
VERTEX_LOCATION = "us-central1" # Live API needs a regional endpoint

Voice Recognition vs. Read Summary Aloud

The two functions use completely different Gemini APIs:

Conclusion

The release of Gemini 3.1 Flash Live makes audio AI more worthy of serious consideration. This time, both voice recognition and read summary aloud were integrated into the LINE Bot:

• Voice Recognition: Standard Gemini API, pre-recorded m4a one-time transcription, connected to the existing Orchestrator
• Read Summary Aloud: Gemini Live TTS, summary text to PCM, ffmpeg to m4a, AudioSendMessage returns

The most troublesome part is not the function itself, but finding the correct model name and locating the SDK’s location logic – neither of these are clearly written in a prominent place in the documentation, and the answer can only be found by listing available models and reading the SDK source code.

The full code is on GitHub, feel free to refer to it.

See you next time!

Abstract

To address the bottlenecks of existing 3D perception technologies, such as high algorithm complexity, exorbitant hardware costs, and insufficient real-time performance, this paper innovatively proposes a lightweight 3D object property recognition method based on camera displacement. By employing a time-sliced focusing strategy and analyzing the objective laws of pixel changes during continuous camera displacement, combined with a dynamic region segmentation strategy along the vertical plumb line, the method achieves rapid identification and 3D modeling of high-convex, low-concave, and planar objects. Its core advantage lies in sharing camera data sources and computational cores with deep learning methods while supporting distributed collaborative operations. Unlike traditional deep learning-based approaches, the proposed method does not require pre-training of object attributes; instead, it determines 3D properties solely through the increasing, decreasing, or offset patterns of pixel differences, significantly reducing computational complexity. Experimental results demonstrate that the method reduces computational latency by approximately 60% while maintaining high recognition accuracy, making it effectively applicable to real-time scenarios such as blind navigation, mobile robotics, and autonomous driving. Additionally, the method can simultaneously resolve visual deception issues like mirror illusions and water reflections, providing a new technical pathway for intelligent perception systems while ensuring data credibility.

Main Contributions

1. A 3D object characteristic recognition method based on pixel variation is proposed, which requires no deep learning models or object attribute training, and directly judges the convex, concave, and planar properties of objects through the increasing/decreasing patterns of pixel differences;
2. A dynamic region segmentation strategy based on the plumb-line direction is designed, which adjusts the division ratios of the left, middle, and right regions in real time according to the camera pose (tilt angle), enhancing the dynamic adaptability of the method;
3. A segmented focusing method is presented, which dynamically adjusts the focus point during continuous camera displacement to ensure the comparability of consecutive frames and improve the accuracy of pixel variation calculation;
4. The pixel offset laws between different regions (left, middle, right) and different object characteristics (convex, concave, planar) are systematically analyzed, and a quantitative relationship between pixel variation and object distance is established;
5. The method can simultaneously solve visual deception problems such as mirror illusions and water surface reflections, providing robust perception capability for unmanned navigation systems.
   # System Architecture and Core Principles
   ## 2.1. System Architecture Overview
   The proposed image acquisition and 3D perception system comprises two primary subsystems:
   hardware and algorithm. The overall architecture is illustrated in Figure 1:
   1 camera, 2 rotating motors, 3 firmware for dual degree of freedom rotating mechanism, 4 gravity
   sensor, 5 gyroscope, 6 fasteners, 7 integrated control center, 8 power supply

Hardware Subsystem

The hardware subsystem consists of the following core components:

1. Monocular Camera: Utilizes CMOS or CCD sensors with dynamic focusing capability. Specifications include 1920×1080 resolution, 30fps frame rate, and 90° Field of View (FOV);
2. Two-Degree-of-Freedom (2-DOF) Rotation Mechanism: Comprises horizontal rotation motors (pan) and vertical rotation motors (tilt), with rotation ranges of ±180° and ±60° respectively, and angular resolution of 0.1°. Flexible camera adjustment is achieved through arc-shaped brackets and connecting rods;
3. Attitude Sensor Module: Integrates gravity sensors and gyroscopes for real-time detection of camera tilt angle and angular velocity, with 100Hz sampling frequency. Sensor data is transmitted to the main control unit via I²C bus;
4. Main Control Unit: Employs ARM Cortex-A72 processor or NVIDIA Jetson Nano, integrating MCU/GPU for pixel difference analysis and real-time decision-making. Supports multi-threaded parallel processing to ensure algorithmic real-time performance;
5. Platform Body: Equipped with battery, motor drive module, and communication interface, enabling integration into mobile robots, assistive navigation devices, or autonomous vehicles.
   ### Algorithm Subsystem
   The algorithm subsystem comprises the following modules:
6. Image Acquisition and Preprocessing Module: Responsible for continuous frame acquisition and preprocessing operations including denoising and normalization;
7. Dynamic Region Segmentation Module: Based on gravity sensor and gyroscope data, calculates the vertical plumb line direction in real-time, dividing the image into left, center, and right regions (Figure 2);

Figure 2. Dynamic Region Segmentation Module.

1. Pixel Change Calculation Module: Compares pixel differences between current and previous frames, extracting and stratifying color differences.
2. Object Feature Recognition Module: Determines object properties (convex, concave, planar) based on pixel change patterns;
3. 3D Modeling and Localization Module: Combines object properties and distance information to construct 3D spatial models and output object positions and contours.

Core Principles

Pixel Change Pattern Analysis

When a camera moves continuously along a straight trajectory, different object types exhibit specific geometric patterns in the image. Let C denote the camera’s initial position, C1 the position after movement, and S the displacement vector. For an object in 3D space, its projection on the image plane changes with camera displacement.

1. Planar Objects: Vertical edges of planar objects undergo angular offset in consecutive frames. Specifically, vertical edges in the left region shift rightward, those in the right region shift leftward, and object height in the center region increases. The offset angle θ is inversely proportional to object distance S (Figure 3):

1. High-Convex Objects: The apex of convex objects remains invariant along the vertical plumb line direction across consecutive frames, while edges undergo parallel offset. Convex objects in the left region offset leftward, those in the right region offset rightward, and object height in the center region decreases. The offset magnitude ∆ p relates to protrusion height h and distance S (Figure 4):

1. Low-Concave Objects: Internal surfaces of concave objects expose new areas as the camera moves. The right edge of concave objects in the left region expands outward, the left edge in the right region expands outward, and object height in the center region increases with emerging internal details (Figure 5).

Dynamic Region Segmentation Strategy

Traditional image segmentation methods employ fixed grid divisions that cannot adapt to camera pose changes. The proposed dynamic region segmentation strategy uses the vertical plumb line detected by gravity sensors as a baseline to divide the image into three regions:
• Center Region (R_center): Centered on the plumb line, extending left and right. The center region occupies a smaller width (typically 5%-10% of image width) and primarily captures depth changes of objects directly ahead.
• Left Region (R_left) and Right Region (R_right): Located to the left and right of the center region, occupying the majority of image width.
When the camera rotates around the Y-axis, its optical axis forms an angle a with the platform’s forward direction. According to projective geometry principles, the width proportions of left, center, and right regions should be dynamically adjusted (Figure 6):

• When a < 90◦C, the right region width increases while the left region width decreases:
• When a > 90◦C, the left region width increases while the right region width decreases:

Segmented Focusing Method

During continuous camera displacement, objects at varying distances successively enter the
focusing range. To ensure inter-frame comparability, this paper proposes a segmented focusing
method (Figure 7):

1. Environmental Feature Analysis: Based on current scene depth distribution, focusing distances are divided into three intervals: near-field (0-2m), mid-field (2-5m), and far-field (5m and beyond).
2. Dynamic Focus Point Adjustment: Within each time segment t, the camera continuously focuses on a specific distance interval. When platform displacement exceeds threshold ∆S, focus switches to the next distance interval.
3. Inter-Frame Association: Focus distance and platform position (x, y) are recorded for each frame to ensure compared frames focus on the same spatial area, avoiding errors from depth-of-field changes.

Marty from VTeacher here!

For AI-based security checks, if you are not working in a monorepo, Claude Code with Agent Team is quite handy. It can discover security holes that span across repositories, such as between the frontend and backend.

• .claude/settings.local.json

{
  "env": {
    "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS": "1"
  }
}

Einstein

At this point, some people may be thinking, “Wait, that Einstein!?” So from here on, I mean the scientist Albert Einstein. Incidentally, the Japanese comedy duo Einstein appears to have no connection to the scientist in the origin of their name.

Einstein’s theory of relativity taught us that time passes differently depending on one’s frame of reference, and that strong gravity distorts space itself. I think this theory can also be applied to the world of code review.

And Einstein’s life cannot be reduced to the phrase “genius physicist.” He spent much of his life confronting discrimination and prejudice. He also deeply regretted the military use of science, and continued to advocate for nuclear disarmament and peace until the end of his life. He was even asked to become President of Israel, but declined — a reflection of just how strong a pacifist he was. If he were to see the confrontation among the United States, Israel, and Iran in 2026, I cannot help but think he would grieve deeply.

1. Einstein on the “disease of division”

When prejudice limits people’s rights, diminishes their voices, and shapes institutions themselves into something unequal, that is not abstract. Einstein experienced it personally.

The problem in review is similar.
Unfortunately, it seems to apply to the relationship between reviewer and reviewee as well.
The issue is not just individual behavior. It is closer to a structural question: whose voice gets discounted, and who is more likely to be put at a disadvantage?

To ground that point, I would first like to introduce a report from Google.

Google report: power dynamics in code review

https://research.google/pubs/the-pushback-effects-of-race-ethnicity-gender-and-age-in-code-review/

According to this report, even Google — a company full of highly skilled technologists — struggles with bias in code review, including bias related to race. The report estimates that this results in more than 1,000 extra engineering hours wasted every day. That is about 4% of the estimated time engineers spend responding to reviewer comments, and the burden is borne primarily by the reviewee.

Code review is vulnerable to human bias,
and outcomes differ by gender, race/ethnicity, and age.
Google is estimated to waste more than 1,000 extra engineering hours per day.

If we organize the research data, it looks roughly like this:

1. Women are more likely to face rejection
   The rate of pushback against women’s opinions or proposals is 1.21 times higher than it is for men.
2. Non-white people are more likely to face rejection
   Compared with white reviewers or reviewees, the rate of pushback is 1.54 times higher for Black people, 1.15 times higher for Hispanic/Latinx people, and 1.42 times higher for Asians.
3. Older people are more likely to face rejection
   As age increases, so does the rate of pushback. For example, at age 60, the rate is more than three times higher than for younger people.

In other words, even when the code itself is not the problem, bias can still make code review take longer.

2. Einstein’s theory of relativity

Einstein discovered that time moves more slowly in places with stronger gravity.

In code review, when a massive gravitational force of prejudice appears — like “Someone like Inada would do that,” or “She must be wrong because she’s a woman” — a check that should take one hour can take several. The worker’s “clock of thought” gets distorted like it has fallen into a bottomless swamp.

So let us treat gravity in review as bias, and try applying Einstein’s general theory of relativity to code review.

• Relativistic kinetic energy

• Time dilation

We use the time-dilation formula derived from the Lorentz transformation.

• △t: the original review time
• △t’: the review time stretched by prejudice
• c (the limiting speed): set to $10$ for this example

• v (relative velocity): defined here as the “relative distance” from the baseline of “young, same-race, male.” The farther away one is, the closer v gets to c, and time stretches toward infinity.

  • v = 1 * a
  • a (bias multiplier)
    • Age gap: (young) 0, (middle-aged) +1.18, (older) +3.0
    • Race gap: (same race) 0, (different race) +1.54
    • Gender gap: (male) 0, (female) +1.21

Parallel worlds

Let us compare two timelines: one with bias, and one without.

• Timeline A: △t — the original review time
  Here, we assume 160 hours (about one month).

• Timeline B: △t’ — the review time stretched by bias
  We assume the code itself is not the problem.

3. Einstein’s “equation for happiness”

Einstein is said to have left behind this playful formula for happiness:

A = x + y + z

• x = Work
• y = Play
• z = Keeping your mouth shut

That z points to another important source of friction in code review.

Here I would like to introduce a recent study on toxic comments in software development.

https://arxiv.org/html/2503.10072v2

The paper identifies the following 11 categories of toxic comments:

1. Vulgarity
2. Insult
3. Unprofessional
4. Entitlement
5. Toxicity Towards Tools
6. Mocking
7. Frustration
8. Profane Technology Naming
9. Identity Attack
10. Threat
11. Arrogant

In particular, the last one — Arrogant — deserves attention. When bias causes delays in code review, this kind of comment seems to appear frequently. For example:

Whatever this function is supposed to achieve, 
it’s very weird and messed up. 
Wouldn’t it make more sense to fix this function like I suggested..

At first glance, some people might think this is still within an acceptable range.
But this is classified as a toxic comment.

Remember Einstein’s “equation for happiness”:

Keeping your mouth shut

Following that principle, if you catch yourself about to write a toxic comment, it is worth rewriting it.

How to replace Arrogant comments

Here are some examples of arrogant comments and how to rewrite them. Try them yourself.

I’m not sure what this module is even trying to do. The flow feels pretty disjointed. Wouldn’t it be safer to follow the steps I already outlined?

* Problem: It puts the other person down through definitive language and evaluative wording, while pressuring them to follow the speaker's preferred solution. There is no concrete evidence for X or Y.
* Better approach: Lead with uncertainty, identify the specific missing point and its impact, and add a proposal with a minimal diff.
* Rewrite: Sorry if I’m misunderstanding the premise, but X seems to be missing from the flow, which may affect Y. Would you be open to trying the approach I shared earlier? I can attach a suggested change.

 This is kind of unique, but also a bit hard to follow. Instead of reinventing the wheel, wouldn’t it be better to stick to the pattern we discussed?

* Problem: It dismisses the implementation with vague evaluative language and relies on “because that’s what everyone does” as its only justification.
* Better approach: Point to an existing referenceable example, make consistency the explicit criterion, and offer implementation support.
* Rewrite: For consistency with the existing pattern (link), would you be open to aligning this structure the same way? If helpful, I can put together a patch.

This naming is strange and hard to understand. Wouldn’t it make more sense to use the name I suggested?

* Problem: It implies a lack of ability or good judgment and gives no clear criterion for why the alternative is better.
* Better approach: Explain the cognitive load, provide a domain-aligned naming criterion, and offer a concrete minimal-diff suggestion.
* Rewrite: The current name makes the intent a bit hard to follow. Would you be open to renaming it to match the domain term X? I added a small inline suggestion.

This state management approach feels unusual and fragile. Wouldn’t applying the standard pattern solve it?

* Problem: It relies on abstract evaluative wording and “because it’s standard” pressure, without stating reproduction steps or concrete impact.
* Better approach: Spell out the reproduction condition and specific risk, then provide a referenceable pattern and a patch.
* Rewrite: In case X, cleanup seems to be missing, which may cause Y. Would you be open to switching to the standard pattern (link)? I can suggest a minimal patch.

I don’t see what value this abstraction adds. It just makes things more complex. Wouldn’t it be wiser to remove it as I suggested?

* Problem: It dismisses the value of the work outright and uses superiority language like “wiser” to pressure the other person.
* Better approach: Point to actual reuse frequency, explain how complexity would decrease, and show a concrete change.
* Rewrite: This layer appears to be used only by component X, so inlining it may reduce complexity. I can include a suggested change as an example.

This error handling is inconsistent with the rest of the code. Doesn’t the pattern I showed earlier make more sense?

* Problem: It criticizes using subjective wording but does not clearly explain the actual difference in behavior.
* Better approach: Show the concrete mismatch (return vs. throw), clarify the benefit of consistency, and attach a reference link or patch.
* Rewrite: Elsewhere we return `{type, message}` (link), but here we `throw`. Would you be open to aligning this with the common return shape? I can attach a patch.

Whatever the intent is, the component boundary here feels odd. Why not restructure it like the example?

* Problem: It offers only value judgment (“odd”) without identifying what is actually wrong.
* Better approach: State the evaluation axis explicitly — such as separation of concerns — and suggest where the split should happen, with a guideline reference if possible.
* Rewrite: View logic and data concerns appear to be mixed at this boundary. Splitting at X may better match the guideline (link). I can sketch an approach inline if useful.

This test feels oddly overengineered for what it is trying to verify. Wouldn’t the simpler version I suggested be more reasonable?

* Problem: It belittles the test using evaluative language like “overengineered,” while leaving the concrete over/under-testing issue vague.
* Better approach: Clarify the intended test scope, identify what is unnecessary, and offer a lighter alternative with a small example.
* Rewrite: Since this test is only intended to verify X, the full environment may not be necessary. Would you be open to switching to a lighter fixture? I attached a minimal example.

How to replace Arrogant comments

As a countermeasure against arrogant review comments, I recommend preparing reusable review comment templates like the following.

GitHub: Saved replies

Reviewers can call them up with a slash command when commenting.
If you review locally, another option is to register them as a Claude Code skill or something similar.

# Saved Replies Catalog (EN)
This file is a shared catalog of review comment templates for the team.
Each person can copy these into GitHub > Settings > Saved replies.
They can also be reused as source text for slash-command bots (/cr XXX) or user scripts.

## How to use
1) Choose the relevant item (such as CR-01) and register the title and body as a Saved Reply
2) Replace placeholders in the form of {{like this}}
3) Keep tone and structure consistent: preface → facts → reason → proposal → reference link → Suggested change if needed

## Style rules
- Placeholders: {{target}}, {{factX}}, {{impactY}}, {{permalink}}, etc.
- Use permalinks whenever possible for reference links
- Avoid commands, definitive statements, and evaluative words (weird / obvious / terrible / etc.)

---

## CR-01 Structured comment (problem / reason / proposal)
Sorry if I’m misunderstanding the premise, but in {{target}} I see {{factX}}, which may affect {{impactY}}.
Reason: {{reason / evidence}} (reproduction: {{steps}})
Proposal: Would you be open to considering {{optionA}} or {{optionB}}?
Reference: {{guideline or existing example permalink}}
If helpful, I can add a Suggested change.

---

## CR-02 Consistency (align with an existing pattern)
This differs from the existing {{patternName}} in an important way (reference: {{permalink}}).
Reason: {{factual difference}} (for example, return type / error handling / naming)
Proposal: Aligning this section with the same pattern may make it easier to understand and maintain.
If you agree, I can suggest a minimal-diff patch.

---

## CR-03 Naming (make the intent readable)
The current name makes the intent a bit hard to follow, so I’d like to suggest a name aligned with {{domainTermX}}.
Reason: Clear intent lowers cognitive load and makes later review and maintenance easier.
Proposal: How about changing {{before}} to {{after}}? I can attach a small inline suggestion.

---

## CR-04 State management (missing cleanup / lifecycle issues)
In {{caseX}}, cleanup appears to be missing, which may lead to {{impactY}}.
Proposal: Would you be open to aligning this with the existing state-management pattern (reference: {{permalink}})?
If helpful, I can attach a minimal-diff Suggested change.

---

## CR-05 Error-handling consistency
Only this section uses {{throw / return}}, while the others use {{returnShape}} (reference: {{permalink}}).
Proposal: Would you be open to aligning this with the common return shape {{example: {type, message}}}? I can attach a patch.

---

## CR-06 Component boundary (separation of concerns)
View and data concerns appear to be mixed at this boundary (for example: {{fact}}).
Proposal: Splitting at {{X}} may better match the guideline ({{permalink}}) and keep the impact smaller.
I can sketch a possible split inline if helpful.

---

## CR-07 Testing (adjust scope)
Since this test is intended only to verify {{goalX}}, {{unnecessaryElementY}} may not be needed.
Proposal: Would you be open to switching to a lighter fixture? I can attach a minimal example (reference: {{permalink}}).

---

## CR-08 Request for supporting references
To move the discussion forward, could you share one concrete supporting reference?
For example: a guideline, a spec, an existing PR permalink, or measurement data (Lighthouse / bundle size / etc.)
I can also look for it, but having it upfront would help us move faster.

---

## CR-09 Request for a guided tour (for large diffs)
Because this diff is broad, could you provide a short guided tour first?
For example, links at the top of the PR in this order:
1) routing, 2) UI/state, 3) API/types, 4) tests
This helps prevent missed context and makes it easier to sync up before detailed comments.

---

## CR-10 Acknowledgment / agreement text (to soften tone)
Thanks for sharing this. I understand the intent. My only concern is {{one-point summary}}.
If we can align on that, the rest looks good to me. I can suggest a minimal diff if useful.

---

## CR-99 Final check before sending
- Are there any evaluative words included (weird / overblown / obvious / etc.)?
- Did I separate facts from impact?
- Did I include a supporting link (permalink / guideline / measurement)?
- Is the proposal concrete (Suggested change / minimal diff)?
- Did I include a short uncertainty preface (Sorry if I’m misunderstanding)?

4. Moving toward peace, like Einstein

The United States, Israel, and Iran are currently in a state of war.

Einstein also had a historical connection to Israel.
In 1952, after the death of Israel’s first president, the Israeli government asked Einstein to become president.
He declined.

The reasons usually given include the following:

1. He believed he lacked the talent to deal with people
2. He did not want to lose his freedom
3. He was an extremely strong pacifist

If Einstein were alive today, perhaps he would ask something like this:

“Will you really keep fighting while continuing to bind nationalism to the logic of nuclear weapons?”

The same applies to review, I think.
We need to look not only at the code, but also at the relationships around it.
And if we want those relationships to become even a little more peaceful, then we need to change the way we write comments, the way approval works, and the culture itself.

I hope your team can become a place that deepens quality quietly, rather than deepening conflict.

Of course, there is no single absolute answer for code review.
Practices vary greatly from team to team.

But if you are starting to feel the limits of human-centered review, it may be worth letting AI take over more of it.

So, in conclusion…

The number-one anti-pattern in human code review that we can learn from Einstein is:

Cognitive bias.

Thanks for reading.
(I’ll save number two and beyond for another time.)