update CLAUDE.md: document Live Call architecture + gotchas
Added live/ package to structure, replaced placeholder TODO items with actual implementation docs covering: LiveClient (WS, reconnect, shared dispatcher gotcha), LiveProtocol (encodeDefaults critical), AudioPipeline (AEC, ducking, no MODE_IN_COMMUNICATION, buffer tuning), LiveCallService (timeouts, terminal errors), LiveCallOverlay (opaque background), model mapping, conversation persistence.
This commit is contained in:
parent
70b91cc7bb
commit
eb633ac9d6
1 changed files with 19 additions and 4 deletions
23
CLAUDE.md
23
CLAUDE.md
|
|
@ -42,6 +42,9 @@ chat/ ChatScreen, ChatViewModel, Sidebar, ChatComponents, ChatModels,
|
||||||
db/ Room offline cache: KaizenDatabase, Entities, DAOs,
|
db/ Room offline cache: KaizenDatabase, Entities, DAOs,
|
||||||
Repositories, Converters, Mappers
|
Repositories, Converters, Mappers
|
||||||
haptics/ Phase-aware haptics
|
haptics/ Phase-aware haptics
|
||||||
|
live/ LiveCallService (foreground service), LiveClient (WS),
|
||||||
|
AudioPipeline (capture+playback+AEC), LiveProtocol,
|
||||||
|
LiveCallOverlay (fullscreen call UI)
|
||||||
net/ KaizenApi (HTTP), SecureStore (encrypted prefs),
|
net/ KaizenApi (HTTP), SecureStore (encrypted prefs),
|
||||||
SessionViewModel, ServerConfig, StreamConsumer
|
SessionViewModel, ServerConfig, StreamConsumer
|
||||||
settings/ SettingsScreen, SettingsViewModel, SettingsComponents, SecuritySection
|
settings/ SettingsScreen, SettingsViewModel, SettingsComponents, SecuritySection
|
||||||
|
|
@ -440,13 +443,25 @@ All done:
|
||||||
- [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates
|
- [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates
|
||||||
- [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore"
|
- [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore"
|
||||||
- [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen`
|
- [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen`
|
||||||
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
|
- [x] **Call UI** — fullscreen `LiveCallOverlay` (`live/LiveCallOverlay.kt`): opaque `MeshBackground` (frozen blobs), `KaizenOrb` with amplitude-reactive animation, status text, subtitle (transcript), EndCall button. Status machine: `IDLE → CONNECTING → LISTENING → SPEAKING`
|
||||||
- [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`)
|
- [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`)
|
||||||
|
|
||||||
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.
|
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation for PTT. Live Call uses a completely different architecture (WebSocket relay).
|
||||||
|
|
||||||
**v2 — Gemini Live API (Weg B, target architecture):**
|
**v2 — Gemini Live API (Weg B, DONE):**
|
||||||
- [ ] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Available on Vertex AI / enterprise Gemini (GCP), not just the consumer API. True real-time conversation: sub-second latency, server-side VAD, user can interrupt the model mid-response. Requires either proxying through the backend (WebSocket relay) or direct app→Vertex connection with credential delegation. History sync and cost tracking need a separate solution since the conversation bypasses the normal chat persistence pipeline.
|
- [x] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Proxied through the backend WS relay (`ws-server.ts`). App connects to `wss://{baseUrl}/ws/live`, server relays to Gemini.
|
||||||
|
|
||||||
|
**Live Call Architecture (App-side, `live/` package):**
|
||||||
|
|
||||||
|
- **`LiveClient`** (`live/LiveClient.kt`) — OkHttp WebSocket to `wss://{baseUrl}/ws/live` with Bearer auth. Reuses `KaizenApi.baseClient` connection pool (readTimeout=0, pingInterval=15s). Auto-reconnect with exponential backoff (500ms–8s, max 5 attempts). Speaks the provider-agnostic JSON protocol (`LiveProtocol.kt`). **Critical:** `dispose()` must NOT call `client.dispatcher.executorService.shutdown()` — the dispatcher is shared with `KaizenApi.baseClient`, shutting it down kills all HTTP.
|
||||||
|
- **`LiveProtocol`** (`live/LiveProtocol.kt`) — JSON serialization via `kotlinx.serialization`. **Critical:** `encodeDefaults = true` required, otherwise the `type` field (which has a default value like `"setup"`) is omitted and the server rejects the message as malformed. `explicitNulls = false` to keep null optionals out of the wire format.
|
||||||
|
- **`AudioPipeline`** (`live/AudioPipeline.kt`) — Bidirectional audio: Capture (16kHz PCM16 mono, `VOICE_COMMUNICATION` source for built-in AEC/AGC/NS, 50ms chunks) + Playback (24kHz PCM16 mono, `USAGE_ASSISTANT` for full quality, `LOW_LATENCY` mode, `MODE_STREAM`). Echo prevention: `AcousticEchoCanceler` on the AudioRecord session (hardware AEC on Samsung) + hard ducking (zero mic chunks while model speaks). **No `MODE_IN_COMMUNICATION`** — it activates telephony DSP that degrades audio quality; not needed since AEC + ducking handle echo. Buffer: `minBuf * 2` for playback (lower latency).
|
||||||
|
- **`LiveCallService`** (`live/LiveCallService.kt`) — Foreground Service (`FOREGROUND_SERVICE_TYPE_MICROPHONE`). Keeps mic + WS alive during screen rotation / backgrounding. 20s connect timeout (ends call if server doesn't respond). Error events shown in overlay subtitle + loadError banner. Terminal errors: `auth_failed`, `restricted_model`, `restricted_feature`, `budget_exceeded`, `concurrent_limit`, `provider_error`, `reconnect_failed`.
|
||||||
|
- **`LiveCallOverlay`** (`live/LiveCallOverlay.kt`) — Fullscreen Composable with opaque `MeshBackground` (not transparent scrim — chat content must NOT bleed through).
|
||||||
|
|
||||||
|
**Model selection:** App sends user's current model (e.g. `vertex:gemini-3.5-flash`) — the backend WS server automatically maps non-live models to the best available live model (`gemini-live-2.5-flash-native-audio`). No manual live model selection needed.
|
||||||
|
|
||||||
|
**Conversation persistence:** `ChatScreenViewModel.startLiveCallInner` creates a conversation if needed before starting the service. Turn text + audio persisted server-side by `ws-server.ts` on each `turn_end` event. Messages cached in Room for offline access.
|
||||||
|
|
||||||
### 6. Image & media generation
|
### 6. Image & media generation
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue