In-app conversational assistant overview

In Production2026

In-App Conversational AI Agent

Italian Telco

A production voice and chat assistant embedded in a major Italian telecom operator's mobile app, letting subscribers manage their line, activate offers, and get support by talking or typing instead of navigating menus. Designed and built the real-time orchestration backend and the entire in-app assistant experience across iOS and Android, end to end.

The operator hasn't launched publicly yet — launch is expected in early Q4 2026 — so this case study is written without the product or client name. It will be updated with the real name and in-app screenshots after launch.

Python

FastAPI

Gemini Live

Google Cloud

MongoDB

Kotlin Multiplatform

SwiftUI

System Architecture

Real-Time Orchestration

A Gemini Live backend built for voice

The assistant runs on a Python/FastAPI service designed and built end to end for this project. The two voice modes — full-duplex speech-to-speech and a push-to-talk mode that records, lets the user review, then sends — run on Gemini's native-audio Live model over a streaming WebSocket; text chat runs a separate multi-step tool-calling loop on standard Gemini. Over 40 tool declarations span sales, account, billing, and support so the model can act, not just answer.

Guardrails & Safety

Screening both sides of the conversation

Every prompt and every model response is screened through Google Cloud Model Armor and DLP before it reaches the user, deliberately fail-open so a guardrail-service outage degrades gracefully instead of taking the assistant down mid-call.

Data & Content Pipeline

Designed and built the FastAPI/MongoDB service behind both the app's catalog screens and the assistant's tool calls — including the cron jobs that keep it in sync. Nothing here is hand-maintained.

Daily sync jobs

Cloud Scheduler cron jobs run daily against the operator's WordPress site and WooCommerce store, scraping and normalizing plans, options, and rates into MongoDB, plus a purpose-built scraper for the pages that have no API at all.

167 FAQs, 15 support pages

The operator's FAQ and consumer-support content is scraped and embedded into a vector-search collection, so the assistant grounds procedural and support answers in the operator's own copy instead of improvising.

One source, two consumers

The native app's catalog screens and the assistant's tool calls read from the exact same endpoints, so a plan that's wrong or stale can never differ between the two.

In the App

Entry Point

A floating orb replaces the chat icon

The assistant lives behind a floating orb rather than a conventional chat icon. A long press opens a pick between talking and typing, and the same entry point works from anywhere in the app on both iOS and Android.

Real-Time Speech-to-Speech

Audio flows both ways, continuously

Voice runs on Gemini's native-audio Live model over a bidirectional WebSocket — audio in, audio out, no intermediate speech-to-text-to-speech hop — so the assistant can be interrupted mid-sentence on both iOS and Android and pick the reply back up as part of the same continuous stream. The production voice itself was chosen by an internal listening panel across several candidates, which picked the one that sounded most natural and expressive, even though it ranked last on the acoustic monotone-ness metric used to screen candidates.

Actions, Not Just Answers

Conversations that complete real tasks

Offer cards surfaced mid-conversation can be activated without leaving the voice session, and a guided top-up flow — rebuilt end to end — is wired to real payment methods with an auto-recharge follow-up on success.

Access Control

Auth-gated by design

The assistant checks real authentication tokens before it can touch anything account-specific, so a logged-out session stays limited to general information only.

Custom Voice Activity Detection

The server's own voice detection can tell that someone is speaking, but not whether it's the actual user — echo, background chatter, or another voice nearby can trigger it just as easily. Designed and built a client-side adaptive noise gate to solve that, first validated on iOS and then ported and independently recalibrated for Android, whose mic path attenuates very differently.

Adaptive ambient floor

An asymmetric moving average — slow to rise, faster to fall — learns the room's noise floor in real time, so the gate's opening threshold scales to how loud the room already is instead of a fixed constant.

Pre-roll buffering

A short rolling buffer of recent audio is retained while the gate is closed and flushed the instant it opens, so the very start of a word is never clipped.

Stuck-open watchdog

If the gate stays open with no recognized speech for too long, it force-closes and reseeds the noise floor from what it just heard, so a noisy room can't leave it stuck wide open.

Hardware AEC + software gate

On loudspeaker playback, the OS's own echo canceller does the first pass and the gate cleans up the rest. An on-device test showed hardware cancellation alone still let the assistant transcribe its own voice as user input: the exact failure the gate was built to fix.

←Previous

Event Companion App