From 3c6dab33a0e0b4a569797b57bb956850412c94e8 Mon Sep 17 00:00:00 2001 From: Bruno Deanoz Date: Tue, 23 Jun 2026 00:10:15 +0200 Subject: [PATCH] docs: add Call Mode roadmap (Weg A pipeline + Weg B Live API), image/media gen section - Mark auto-logout on 401 as done - Remove WebSocket text-streaming TODO (not needed) - Add Voice & Call Mode section with v1 (pipeline) and v2 (Gemini Live API via Vertex/GCP) - Add Image & media generation section - Document stop button, locale fix, login shadow migration in shipped features --- CLAUDE.md | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d6623a4..f1d2110 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -259,6 +259,7 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo | `70aa5c5` | **ChatInput button layout** — Mic (voice) + Plus left, Call/Send toggle right. AudioLines icon for call. SendButton: no border, translucent gradient, 38dp | | `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled | | `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation | +| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow | **Earlier UI/feel work (Phase 0 prototype → feel-first):** - Liquid glass styling, floating panels, glassmorphic sidebar @@ -319,7 +320,7 @@ These are what separate "prototype" from "daily-driver": ### 3. Security hardening -- [ ] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner +- [x] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner - [ ] **Server version check** — call `/api/v1/meta` after login to detect server skew and warn cleanly - [x] **Biometric lock** (`BiometricPrompt` + KeyStore/CryptoObject) — hardware-enforced, password fallback dialog when unavailable, biometric toggle in Settings, all unlock paths gated - [x] **Security settings** — biometric toggle, signed-in devices with remote revoke, password change @@ -334,17 +335,35 @@ These are what separate "prototype" from "daily-driver": - [x] **Web search sources** — `SourcesBlock` (collapsible, clickable links via ACTION_VIEW), parses `\x02`/`\x04` sentinels. `webSearch: true` in ChatRequest triggers agentive search - [x] **Tool call display** — `ToolEventsBlock`, parses `\x03` sentinel, status icons per step - [ ] **Native passkey login** (WebAuthn/FIDO2 on Android) -- [ ] **WebSocket streaming** (replace chunked HTTP for lower latency on mobile) - [ ] **Usage/cost display** - [ ] **User context / KI-Profil** -### 5. App distribution +### 5. Voice & Call Mode + +**v1 — Pipeline (Weg A, uses existing backend):** +- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively) +- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer` +- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices) +- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen` + +The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed. + +**v2 — Gemini Live API (Weg B, target architecture):** +- [ ] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Available on Vertex AI / enterprise Gemini (GCP), not just the consumer API. True real-time conversation: sub-second latency, server-side VAD, user can interrupt the model mid-response. Requires either proxying through the backend (WebSocket relay) or direct app→Vertex connection with credential delegation. History sync and cost tracking need a separate solution since the conversation bypasses the normal chat persistence pipeline. + +### 6. Image & media generation + +- [ ] **Image generation** — send prompt to `/api/image-gen` (needs Bearer auth + v1 re-export), display returned image URL inline. Web uses `detectMode("image")` + dedicated `imageModelId` picker; app can start simpler (Bild mode pill → image-gen endpoint) +- [ ] **Video generation** — `/api/video-gen` (SSE polling, max 4.5 min), display result inline +- [ ] **Audio generation** — music (Lyria) + speech (Gemini-TTS) via `/api/audio-gen`, inline `