diff --git a/CLAUDE.md b/CLAUDE.md index 1b8e48e..700383e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -42,6 +42,9 @@ chat/ ChatScreen, ChatViewModel, Sidebar, ChatComponents, ChatModels, db/ Room offline cache: KaizenDatabase, Entities, DAOs, Repositories, Converters, Mappers haptics/ Phase-aware haptics +live/ LiveCallService (foreground service), LiveClient (WS), + AudioPipeline (capture+playback+AEC), LiveProtocol, + LiveCallOverlay (fullscreen call UI) net/ KaizenApi (HTTP), SecureStore (encrypted prefs), SessionViewModel, ServerConfig, StreamConsumer settings/ SettingsScreen, SettingsViewModel, SettingsComponents, SecuritySection @@ -440,13 +443,25 @@ All done: - [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates - [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore" - [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen` -- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices) +- [x] **Call UI** — fullscreen `LiveCallOverlay` (`live/LiveCallOverlay.kt`): opaque `MeshBackground` (frozen blobs), `KaizenOrb` with amplitude-reactive animation, status text, subtitle (transcript), EndCall button. Status machine: `IDLE → CONNECTING → LISTENING → SPEAKING` - [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`) -The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed. +The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation for PTT. Live Call uses a completely different architecture (WebSocket relay). -**v2 — Gemini Live API (Weg B, target architecture):** -- [ ] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Available on Vertex AI / enterprise Gemini (GCP), not just the consumer API. True real-time conversation: sub-second latency, server-side VAD, user can interrupt the model mid-response. Requires either proxying through the backend (WebSocket relay) or direct app→Vertex connection with credential delegation. History sync and cost tracking need a separate solution since the conversation bypasses the normal chat persistence pipeline. +**v2 — Gemini Live API (Weg B, DONE):** +- [x] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Proxied through the backend WS relay (`ws-server.ts`). App connects to `wss://{baseUrl}/ws/live`, server relays to Gemini. + +**Live Call Architecture (App-side, `live/` package):** + +- **`LiveClient`** (`live/LiveClient.kt`) — OkHttp WebSocket to `wss://{baseUrl}/ws/live` with Bearer auth. Reuses `KaizenApi.baseClient` connection pool (readTimeout=0, pingInterval=15s). Auto-reconnect with exponential backoff (500ms–8s, max 5 attempts). Speaks the provider-agnostic JSON protocol (`LiveProtocol.kt`). **Critical:** `dispose()` must NOT call `client.dispatcher.executorService.shutdown()` — the dispatcher is shared with `KaizenApi.baseClient`, shutting it down kills all HTTP. +- **`LiveProtocol`** (`live/LiveProtocol.kt`) — JSON serialization via `kotlinx.serialization`. **Critical:** `encodeDefaults = true` required, otherwise the `type` field (which has a default value like `"setup"`) is omitted and the server rejects the message as malformed. `explicitNulls = false` to keep null optionals out of the wire format. +- **`AudioPipeline`** (`live/AudioPipeline.kt`) — Bidirectional audio: Capture (16kHz PCM16 mono, `VOICE_COMMUNICATION` source for built-in AEC/AGC/NS, 50ms chunks) + Playback (24kHz PCM16 mono, `USAGE_ASSISTANT` for full quality, `LOW_LATENCY` mode, `MODE_STREAM`). Echo prevention: `AcousticEchoCanceler` on the AudioRecord session (hardware AEC on Samsung) + hard ducking (zero mic chunks while model speaks). **No `MODE_IN_COMMUNICATION`** — it activates telephony DSP that degrades audio quality; not needed since AEC + ducking handle echo. Buffer: `minBuf * 2` for playback (lower latency). +- **`LiveCallService`** (`live/LiveCallService.kt`) — Foreground Service (`FOREGROUND_SERVICE_TYPE_MICROPHONE`). Keeps mic + WS alive during screen rotation / backgrounding. 20s connect timeout (ends call if server doesn't respond). Error events shown in overlay subtitle + loadError banner. Terminal errors: `auth_failed`, `restricted_model`, `restricted_feature`, `budget_exceeded`, `concurrent_limit`, `provider_error`, `reconnect_failed`. +- **`LiveCallOverlay`** (`live/LiveCallOverlay.kt`) — Fullscreen Composable with opaque `MeshBackground` (not transparent scrim — chat content must NOT bleed through). + +**Model selection:** App sends user's current model (e.g. `vertex:gemini-3.5-flash`) — the backend WS server automatically maps non-live models to the best available live model (`gemini-live-2.5-flash-native-audio`). No manual live model selection needed. + +**Conversation persistence:** `ChatScreenViewModel.startLiveCallInner` creates a conversation if needed before starting the service. Turn text + audio persisted server-side by `ws-server.ts` on each `turn_end` event. Messages cached in Room for offline access. ### 6. Image & media generation